search

Azure / static-web-apps

Azure Static Web Apps. For bugs and feature requests, please create an issue in this repo. For community discussions, latest updates, kindly refer to the Discussions Tab. To know what's new in Static Web Apps, visit https://aka.ms/swa/ThisMonth
https://aka.ms/swa
MIT License
330 stars 57 forks source link

Build and deploy failure trying to start process '/ (deleted)' while restoring projects #755

Closed rbergen closed 2 years ago

rbergen commented 2 years ago

Describe the bug

Since today, the Build and deploy step suddenly started failing during the Oryx restore projects step on a .NET 6 Blazor WASM solution that built fine up to and including yesterday. No changes were made to the workflow file between the last successful and the first failed build.

The error occurs during the project restore step. The relevant snippet of the log is this:

Successfully installed workload(s) wasm-tools.

  Determining projects to restore...
MSBUILD : error MSB1025: An internal failure occurred while running MSBuild.
Microsoft.Build.BackEnd.NodeFailedToLaunchException: An error occurred trying to start process '/ (deleted)' with working directory '/github/workspace/src/Frontend'. No such file or directory
 ---> System.ComponentModel.Win32Exception (2): An error occurred trying to start process '/ (deleted)' with working directory '/github/workspace/src/Frontend'. No such file or directory

What occurs to me as odd is that the inner exception is a Win32Exception, while the runner is ubuntu-latest. Then again, that may just be due to my limited knowledge of the inner workings of the relevant part of .NET.

To Reproduce Steps to reproduce the behavior:

  1. Run a Azure/static-web-apps powered build of the solution

The repository the error occurs with, is: https://github.com/rbergen/PrimeView The GitHub Actions file is: https://github.com/rbergen/PrimeView/blob/staging/.github/workflows/azure-static-web-apps-agreeable-mud-0b27ba210.yml The issue started occurring after opening this PR: https://github.com/rbergen/PrimeView/pull/11

Expected behavior A successful GitHub Actions run.

Log file I've attached the full logs of a failing build to this issue: 1_Build and Deploy Job (1).txt

loganmarshall1 commented 2 years ago

Happening to me to on all my .Net 6.0.1 apps failing in Azure Devops Pipeline, same message. This started happening for us. March 8th 2022

Unhandled exception: Microsoft.Build.BackEnd.NodeFailedToLaunchException: An error occurred trying to start process '/ (deleted)' with working directory '/working_dir/AppMain'. No such file or directory ---> System.ComponentModel.Win32Exception (2): An error occurred trying to start process '/ (deleted)' with working directory '/working_dir/AppMain'. No such file or directory at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) at System.Diagnostics.Process.Start() at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.LaunchNode(String msbuildLocation, String commandLineArgs) --- End of inner exception stack trace --- at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary2 globalProperties, Dictionary2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, TextWriter targetsWriter, Boolean detailedSummary, ISet1 warningsAsErrors, ISet1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, Boolean isolateProjects, GraphBuildOptions graphBuildOptions, Boolean lowPriority, String[] inputResultsCaches, String outputResultsCache) at Microsoft.Build.CommandLine.MSBuildApp.Execute(String[] commandLine) at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args) at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)

loganmarshall1 commented 2 years ago

@anthonychu Hi there my friend. was anything updated recently? in Oryx, in .Net 6.0.101 vs 6.0.201 ? Azure Static Web App Pipeline Task, etc? I have a feeling that hundreds/thousands of pipelines for customers aren't running and this could be wide spread. Thanks my friend and please let us know asap. :) thanks

anthonychu commented 2 years ago

Possibly related to #753

@william-msft can you please look in this as well?

vnbaaij commented 2 years ago

Happening here as well: /opt/dotnet/6.0.100/sdk/6.0.100/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/ (deleted)" is invalid.

It worked fine a couple of hours ago...

ramseytm commented 2 years ago

We're experiencing the same exception with our Azure Dev Ops pipelines as listed by vnbaaij. We're utilizing the AzureStaticWebApp@0 task. If it helps, it started occurring on our end on March 8th (Tuesday). At the time, I did a rerun failed tasks and it worked just fine. As of this morning, March 10th, I cannot get it to succeed and I receive the same error each time.

IEvangelist commented 2 years ago

I'm seeing this too. Here is my error log, for my Blazor WebAssembly app:

https://github.com/IEvangelist/learning-blazor/runs/5499631398?check_suite_focus=true

Unhandled exception: Microsoft.Build.BackEnd.NodeFailedToLaunchException: An error occurred trying to start process '/ (deleted)' with working directory '/github/workspace/src/Web.Client'. No such file or directory
 ---> System.ComponentModel.Win32Exception (2): An error occurred trying to start process '/ (deleted)' with working directory '/github/workspace/src/Web.Client'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.LaunchNode(String msbuildLocation, String commandLineArgs)
   --- End of inner exception stack trace ---
   at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary`2 globalProperties, Dictionary`2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, TextWriter targetsWriter, Boolean detailedSummary, ISet`1 warningsAsErrors, ISet`1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, Boolean isolateProjects, GraphBuildOptions graphBuildOptions, Boolean lowPriority, String[] inputResultsCaches, String outputResultsCache)
   at Microsoft.Build.CommandLine.MSBuildApp.Execute(String[] commandLine)
   at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
   at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
softcraftsman commented 2 years ago

Seeing this too:

Determining projects to restore... Restored /github/workspace/src/api/api.csproj (in 4.02 sec).

Publishing to directory /bin/staticsites/ss-oryx/api...

Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET Copyright (C) Microsoft Corporation. All rights reserved.

Determining projects to restore... All projects are up-to-date for restore. /opt/dotnet/6.0.100/sdk/6.0.100/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/ (deleted)" is invalid. [/github/workspace/src/api/api.csproj]

---End of Oryx build logs--- Oryx has failed to build the solution.

dustuuvr commented 2 years ago

Same issue happening for me:

Determining projects to restore...
  Restored /github/workspace/Client/Client.csproj (in 2.75 sec).
  Restored /github/workspace/Data/Data.csproj (in 2.75 sec).

Publishing to directory /bin/staticsites/ss-oryx/app...

Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  All projects are up-to-date for restore.
/opt/dotnet/6.0.100/sdk/6.0.100/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/ (deleted)" is invalid. [/github/workspace/Data/Data.csproj]

---End of Oryx build logs---
Oryx has failed to build the solution.
mdecaire commented 2 years ago

Same:
Determining projects to restore... All projects are up-to-date for restore. /opt/dotnet/5.0.403/sdk/5.0.403/Sdks/Microsoft.NET.Sdk.Razor/build/netstandard2.0/Microsoft.NET.Sdk.Razor.Component.targets(107,5): error MSB6004: The specified task executable location "/ (deleted)" is invalid. [/working_dir/MailTracking.App/MailTracking.App.csproj]

loganmarshall1 commented 2 years ago

Same: Determining projects to restore... All projects are up-to-date for restore. /opt/dotnet/5.0.403/sdk/5.0.403/Sdks/Microsoft.NET.Sdk.Razor/build/netstandard2.0/Microsoft.NET.Sdk.Razor.Component.targets(107,5): error MSB6004: The specified task executable location "/ (deleted)" is invalid. [/working_dir/MailTracking.App/MailTracking.App.csproj]

@anthonychu @william-msft would this mean dotnet 5 & 6 are both effected?

thanks kindly.

johnpeeke commented 2 years ago

Same issue here, using .net 6.

/opt/dotnet/6.0.100/sdk/6.0.100/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/ (deleted)" is invalid.

william-msft commented 2 years ago

Acknowledged. Thank you everyone for the additional information. We'll be taking a look.

wuilliam321 commented 2 years ago

Same here! same message! 

    <TargetFramework>net6.0</TargetFramework>

Edited:

/opt/dotnet/6.0.100/sdk/6.0.100/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/ (deleted)" is invalid. [/github/workspace/Data/Data.csproj]
loganmarshall1 commented 2 years ago

Hi Everyone,

Just in case anyone is using dotnet 6.0.x or higher, I got mine working by doing the build myself instead of auto building with Oryx. Please see below YML file for blazor wasm dotnet 6.0.x for use with Azure DevOps build pipeline with release to azure static web app. hope that helps!

  1. I had to make a .sln file with ONLY my front end project/dependencies, below its "AppMain.sln"
  2. Since dotnet publish will also build first below is a one liner which feel free to adapt to your specific YML CI CD flows.

`variables: buildConfiguration: 'Release'

csharpfritz commented 2 years ago

This only happens for me with the ubuntu-20 runner. I stepped down to the ubuntu-18 runner and everything ran just fine

loganmarshall1 commented 2 years ago

This only happens for me with the ubuntu-20 runner. I stepped down to the ubuntu-18 runner and everything ran just fine

Thanks @csharpfritz ! do you mind posting the actual code change to yml (or similar) to better support the community during this outage? thanks kindly.

alepauly commented 2 years ago

hey all, thanks for all the information. We're rolling back the Ubuntu 20.04 image to the previous version while we continue to find a root cause. The rollback can take a couple of hours to propagate, I'll confirm here when it's done.

StacyCash commented 2 years ago

This only happens for me with the ubuntu-20 runner. I stepped down to the ubuntu-18 runner and everything ran just fine

Thanks @csharpfritz ! do you mind posting the actual code change to yml (or similar) to better support the community during this outage? thanks kindly.

For GitHub you need to change the 'runs-on' of your job to point to a specific version as in the snippet below

build_and_deploy_job:
    if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.action != 'closed')
    needs: test_job
    runs-on: ubuntu-18.04
wuilliam321 commented 2 years ago 
App Directory Location: 'Client' was found.
Api Directory Location: 'Api' was found.
Looking for event info
The content server has rejected the request with: InternalServerError
Reason: An unexpected error has occurred. Please try again later

With runs-on: ubuntu-18.04

cormacpayne commented 2 years ago

This only happens for me with the ubuntu-20 runner. I stepped down to the ubuntu-18 runner and everything ran just fine

I was cross-checking this with a few of the repositories linked in this thread (sample size of two 😁), and the version of the Ubuntu runner used in the GitHub Action runs seems to be the only thing differing between the latest runs that succeed and fail. Things such as the version of Oryx used, the platform tooling pulled, etc. have not changed according to the logs provided in the runs.

From @rbergen's repository:

Successful run (March 9th, 2022 @ ~2AM PST): 

Environment: ubuntu-20.04
Version: 20220227.1

Failed run (March 10th, 2022 @ ~1pm PST):

Environment: ubuntu-20.04
Version: 20220306.1

From @IEvangelist's repository:

Successful run (March 9th, 2022 @ ~9PM PST): 

Environment: ubuntu-20.04
Version: 20220227.1

Failed run (March 10th, 2022 @ ~9AM PST):

Environment: ubuntu-20.04
Version: 20220306.1
anthonychu commented 2 years ago

Thanks @csharpfritz and others for identifying the workaround.

Please temporarily update your workflow files to use the ubuntu-18.04 hosted runner.

...

jobs:
  build_and_deploy_job:
    if: github.event_name == 'push' || (github.event_name == 'pull_request' && github.event.action != 'closed')
-   runs-on: ubuntu-latest
+   runs-on: ubuntu-18.04
    name: Build and Deploy Job

...

We'll continue to investigate the problem. @william-msft please update this thread if you have more information.

william-msft commented 2 years ago

Thanks @anthonychu, Oryx findings do demonstrate this is related to the environment. If we find additional information we will share.

rbergen commented 2 years ago

Hi @anthonychu, is the rollback of the 20.04 image that @alepauly mentioned still taking place?

anthonychu commented 2 years ago

Hey @alepauly can you please explain what's being rolled back?

alepauly commented 2 years ago

Hey @alepauly can you please explain what's being rolled back?

@anthonychu - the images for the Ubuntu 20.04 hosted agents in Azure Pipelines and runners in GitHub Actions. Since there were enough reports that indicated the virtual environment had unintended breaking changes we decided to rollback while we investigate what's happening. We build those environments from here: https://github.com/actions/virtual-environments

alepauly commented 2 years ago

hey all, thanks for all the information. We're rolling back the Ubuntu 20.04 image to the previous version while we continue to find a root cause. The rollback can take a couple of hours to propagate, I'll confirm here when it's done.

Most (~90%) of the Ubuntu 20.04 runners should be back at version 20220227.1 now. There are still some in the process of rolling back and will continue to do so in the next day. We'll investigate this issue before the next rollout. In the future, any issues related to the virtual environments can be reported at https://github.com/actions/virtual-environments. Thanks!

rbergen commented 2 years ago

I can confirm that I was just able to successfully build the solution again using ubuntu-latest, by retriggering the job. Logging indeed reports the virtual environment version as 20220227.1, again.

Many thanks to everyone who helped in getting this working again on ubuntu-latest. @alepauly and @anthonychu, do let me know if I can be of any further assistance, going forward.

miwebst commented 2 years ago

@alepauly thanks for the info! Is there any way for us to view when the runners change images?

mgrosen commented 2 years ago

@rbergen Thanks for validating that version you are seeing! We have finished rolling back all Ubuntu 20.04 runners, and won't rollout a new version until we validate that this scenario is fixed.

Is there any way for us to view when the runners change images?

The only way I'm aware of is by viewing the Available Environments section of the Readme in the actions/virtual-environments repo.

rbergen commented 2 years ago

Closing this, as the problem that triggered the opening of the issue no longer occurs.