502.5 - Process Failure 0x80004005 Azure .NET 4.6

[odinnix]

ASP.Net MVC Core Application running on .NET 4.6 (full framework). Everything works locally, but when publishing to Azure via one click publish the following error occurs with the standard 502.5 - Process Failure message:

Failed to start process with commandline '"D:\home\site\wwwroot\WebUI-New.exe" ', ErrorCode = '0x80004005'.

I've enabled STDOut logging, it creates the log files - however they are empty.

Originally I thought this was related to 1.0.1 however I've rolled back to 1.0.0 and still get the same error. I've tried everything here:

https://github.com/aspnet/IISIntegration/issues/252

252 seems to be directly related to DNX though.

and here:

https://docs.asp.net/en/latest/publishing/iis.html

The failed request trace is a standard CGI Failure message:

HTTP Error 502.5 - Bad Gateway The specified CGI application encountered an error and the server terminated the process. Most likely causes: The CGI application did not return a valid set of HTTP errors.A server acting as a proxy or gateway was unable to process the request due to an error in a parent gateway.

Project.JSON (v.1.0.0 version however the 1.0.1 version was the same w/ different package versions):

{
    "userSecretsId": "redacted",
    "dependencies": {
        "Autofac.Extensions.DependencyInjection": "4.0.0",
        "Autofac.Extras.CommonServiceLocator": "4.0.0",
        "CommonServiceLocator": "1.3.0",
        "EntityFramework": "6.1.3",
        "Hangfire.AspNetCore": "1.6.4",
        "Hangfire.Autofac": "2.2.0",
        "Hangfire.Core": "1.6.4",
        "Hangfire.SqlServer": "1.6.4",
        "ISO3166": "1.0.1",
        "Microsoft.AspNetCore.Authentication.Facebook": "1.0.0",
        "Microsoft.AspNetCore.Authentication.Google": "1.0.0",
        "Microsoft.AspNetCore.Diagnostics": "1.0.0",
        "Microsoft.AspNetCore.Identity": "1.0.0",
        "Microsoft.AspNetCore.Mvc": "1.0.0",
        "Microsoft.AspNetCore.Razor.Tools": {
            "version": "1.0.0-preview2-final",
            "type": "build"
        },
        "Microsoft.AspNetCore.ReactServices": "1.0.0-beta-000013",
        "Microsoft.AspNetCore.Server.IISIntegration": "1.0.0",
        "Microsoft.AspNetCore.Server.Kestrel": "1.0.0",
        "Microsoft.AspNetCore.SpaServices": "1.0.0-beta-000013",
        "Microsoft.AspNetCore.StaticFiles": "1.0.0",
        "Microsoft.Azure.KeyVault.Core": "1.0.0",
        "Microsoft.Bcl": "1.1.10",
        "Microsoft.Bcl.Build": "1.0.21",
        "Microsoft.Extensions.Configuration.CommandLine": "1.0.0",
        "Microsoft.Extensions.Configuration.EnvironmentVariables": "1.0.0",
        "Microsoft.Extensions.Configuration.Json": "1.0.0",
        "Microsoft.Extensions.Configuration.UserSecrets": "1.0.0",
        "Microsoft.Extensions.Logging": "1.0.0",
        "Microsoft.Extensions.Logging.Console": "1.0.0",
        "Microsoft.Extensions.Logging.Debug": "1.0.0",
        "Microsoft.Extensions.Options": "1.0.0",
        "Microsoft.Extensions.Options.ConfigurationExtensions": "1.0.0",
        "Migrator.EF6.Tools": {
            "version": "1.0.1",
            "type": "build"
        },
        "MR.AspNet.Identity.EntityFramework6": "2.0.0",
        "Newtonsoft.Json": "9.0.1",
        "System.Linq": "4.1.0",
        "WindowsAzure.Storage": "7.2.1"
    },
    "tools": {
        "Microsoft.AspNetCore.Razor.Tools": "1.0.0-preview2-final",
        "Microsoft.Extensions.SecretManager.Tools": "1.0.0-preview2-final",
        "Microsoft.AspNetCore.Server.IISIntegration.Tools": "1.0.0-preview2-final",
        "Microsoft.DotNet.Watcher.Tools": "1.0.0-preview2-final",
        "Migrator.EF6.Tools": {
            "version": "1.0.1",
            "imports": "portable-net45+win8+dnxcore50"
        }
    },
    "frameworks": {
        "net46": {
            "dependencies": {
                "Authentication": {
                    "target": "project"
                },
                "Common": {
                    "target": "project"
                },
                "Domain": {
                    "target": "project"
                },
                "EFStorageProvider": {
                    "target": "project"
                },
                "Models": {
                    "target": "project"
                },
                "Services": {
                    "target": "project"
                }
            }

        }
    },
    "buildOptions": {
        "emitEntryPoint": true,
        "preserveCompilationContext": true
    },

    "runtimeOptions": {
        "configProperties": {
            "System.GC.Server": true
        }
    },

    "publishOptions": {
        "include": [
            ".babelrc",
            "app.config",
            "appsettings.json",
            "App",
            "node_modules",
            "typings",
            "Views",
            "tsconfig.json",
            "tsd.json",
            "web.config",
            "webpack.*.js",
            "wwwroot"
        ]
    },

    "scripts": {
        "prepublish": [
            "npm install"
            "node node_modules/webpack/bin/webpack.js --config webpack.config.vendor.js",
            "node node_modules/webpack/bin/webpack.js"
        ],
        "postpublish": ["dotnet publish-iis --publish-folder %publish:OutputPath% --framework %publish:FullTargetFramework%"]
    },

    "tooling": {
        "defaultNamespace": "WebUI"
    }
}

Post deployment web.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!--
    Configure your application settings in appsettings.json. Learn more at https://go.microsoft.com/fwlink/?LinkId=786380
  -->
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <aspNetCore processPath=".\WebUI-New.exe" arguments="" stdoutLogEnabled="true" forwardWindowsAuthToken="false" stdoutLogFile="\\?\%home%\LogFiles\stdout" />
    <staticContent>
      <remove fileExtension=".ts" />
      <mimeMap fileExtension=".ts" mimeType="application/x-typescript" />
      <remove fileExtension=".tsx" />
      <mimeMap fileExtension=".tsx" mimeType="text/plain" />
    </staticContent>
  </system.webServer>
</configuration>
<!--ProjectGuid: 619bb83b-97e0-4564-bfde-de5493e6163c-->

Program.cs

    public class Program
    {
        public static void Main(string[] args)
        {
         var config = new ConfigurationBuilder()
             .AddCommandLine(args)
             .AddEnvironmentVariables(prefix: "ASPNETCORE_")
             .Build();

         var host = new WebHostBuilder()
                .UseConfiguration(config)
                .UseKestrel()
                .UseContentRoot(Directory.GetCurrentDirectory())
                .UseIISIntegration()
                .UseStartup<Startup>()
                .Build();

            host.Run();
        }
    }

Other Notes

• I've went as far as disabling most of the application by stripping program/startup.cs to the bare requirements
• The solution has ~15 or so projects. Most of them are legacy csprojs
• This was previously deployed and working a while back.
• I've blown out the server and nuget multiple times
• Use the console and running the EXE directly from KUDU shows no errors.

[muratg]

When you run it locally, do you run Kestrel directly, or behind IIS?

[odinnix]

@muratg Behind IIS Express using VS 2015 "run".

I did just try a dotnet run and got a different exception. I will research that a bit and report back.

Edit

dotnet run works fine once I release the port :)

Going to push to local IIS and try as well.

[odinnix]

Two more things:

• Using dotnet publish and IIS under Windows 10 x64 works fine (local computer)
• I published to a brand new azure website just to see, same result.

[muratg]

Thanks @dcarl1. A couple more questions

• How do you deploy to Azure? Kudu (git) or WebDeploy (VS)?
• If you're doing WebDeploy based deployment, can you make sure you're pushing 32-bit app? Azure App Service currently only supports 32-bit apps.

[odinnix]

@muratg Thank you for the help!

I'm deploying via WebDeploy (VS). So I went down the 64/32bit road already a bit - it was indeed deploying a 64bit app, but I swapped the Azure setting to run 64bit apps - does this not work?

Anyways, I got it to deploy as a 32bit application:

Project.JSON changes

    "buildOptions": {
        "emitEntryPoint": true,
        "preserveCompilationContext": true,
        "platform": "x86" <-- new 
    },

    "runtimes": {
        "win7-x64": {},
        "win7-x86": {}
    },

This allowed me to select 'win7-x86' as a target runtime in the publish profile - however now the deployment failed with a file not found error (it was building into win7-x64 still).

I did some digging and installed the X86 .NET Core libs and SDK along with changing my system PATH so they get picked up over the 64 bit version. That got the deployment to work - however I'm still getting the same behavior on Azure.

So, I deployed the 32bit version to my local IIS using dotnet publish --framework win7-x86. The website works - even though the app pool is not configured to allow 32bit apps.

Confirmed its 32bit as well:

Dump of file WebUI-New.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)

[muratg]

I recommend trying to remove win7-x64 from the runtimes section.

65 bit ASP.NET Core apps are not supported on Azure Web Sites as of yet (even if you configure it on the portal.) You shouldn't need to deploy .NET Core libs or SDK -- they're already installed on the server.

So recommendation -- make your app 32bit only, make sure it builds locally, and deploy.

Let me know how it goes!

[moozzyk]

65 bit ASP.NET Core apps are not supported

This is definitely true :trollface:

[odinnix]

@muratg Just gave that a shot with no change. Work's locally but not on the server.

I'm starting to think this is related to something specific to our app rather then a config issue.

The first thing that comes to mind is app.config. We have a service reference in a different project that is expecting an app.config for its endpoint configuration. What are the chances that is causing Azure issues? If it cannot read the config file?

Is the error code 0x80004005 100% related to config? I can't find and documentation about what that code means by itself.

I think the next step should be systematically removing pieces of the application until it boots up...

[muratg]

@pan-wang do you know what this particular error code is?

[pan-wang]

This 502.5 error means the service cannot started. Does you app read some config file itself? Could you please check whether the file was deployed to the right folder? The .exe must be in the same folder as the web.config per your web.config file.

[odinnix]

So... this was just a dumb little error (as most things that take several days to fix are).

In the app.config I had this line:

  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.2" />
  </startup>

Which override everything but only if I rebuilt the entire project.

I removed the line and it works.

[muratg]

Glad to hear!