[Discussion] .NET Core 2.1.3 deployment to App Service

[davidebbo]

Thread to discuss the deployment of 2.1.3 to App Service. Related to https://github.com/Azure/app-service-announcements/issues/130.

[davidebbo]

Currently, 2.1.3 is not deployed on App Service. We are still discussing the timeline for this.

[niemyjski]

This bit me this week. I wish the rollout of security/feature releases were released on azure the same day.

[skyflyer]

I was getting cryptic 502.5 with not many helpful hints in the log file. I decided to try "SCD" deployment and it worked. If anyone else wants to run the latest bits of ASP.NET Core (2.1.3) on Azure, just use dotnet publish -r win-x64 and it should work.

What's puzzling me is that the target framework is still netcoreapp2.1 in the project file... so in essence, it should be compatible. I know this is (a bit) OT, but if some of the MS gurus (@davidebbo hint hint :)) wants to explain, I'm all ears.

[davidebbo]

@skyflyer sorry, I'm on the App Service side, so I can't answer the target fx behavior with any authority. :)

We are making progress toward starting this release, but there are still a few steps to take first.

[davidebbo]

@natemcmaster can you help explain the behavior? Since all the csproj has is netcoreapp2.1, what determines what exact version it will use?

[natemcmaster]

what determines what exact version it will use

The package version of Microsoft.AspNetCore.App represents a minimum runtime version. I recommend you keep your PackageReference version of Microsoft.AspNetCore.App as low as possible and let runtime rollforward do its magic. I realize this has been a continual pain for our users. I am working on documentation to explain better what is happening. We are also planning changes to this in .NET Core 3.0 which should make this problem go away.

As far as same day rollout, @davidebbo and I have been working on improving our process to get the runtime to the Azure team faster. I know you don't come here for excuses, but we recently had a minor staff change and are trying to fill in the gaps left by that change.

[davidebbo]

@natemcmaster from my testing, both VS and dotnet new generate apps that just have <PackageReference Include="Microsoft.AspNetCore.App" />, not specifying a version. But if that's the case, I'm puzzled why so many users appear to be broken when running on App Service which has 2.1.1. Are they all creating apps that hard code a reference to Microsoft.AspNetCore.App 2.1.3?

[natemcmaster]

Many users do not use the versionless package reference to Microsoft.AspNetCore.App. There are a combination of reasons for this - ignorance, bad tooling, a confusing design, etc. Those who have a version on their package reference will often fall into this trap because the NuGet UI offers an update to the package, things may work locally, but when they deploy, the app fails.

[martincostello]

+1 to the above: it's something myself and my team (even with our own VMs where we control the runtime updates) have been bitten by in the past and have learned through experience. We make sure that our deployment targets all have the new runtime installed on them before we bump our package versions to pick up the latest framework version.

In the case of App Service, this leads to me periodically opening up Kudu and running dotnet --version to see whether it's safe to deploy the latest version to prod yet.

[davidebbo]

We make sure that our deployment targets all have the new runtime installed on them before we bump our package versions to pick up the latest framework version

I think the idea is that you should not specify the version at all, so there is nothing to bump. And as soon as App Service gets the new 2.1.x version, you will automatically start using it. So things should much less painful that you are making it be :)

@natemcmaster please confirm that this is correct, and that be not specifying anything, you are guaranteed to run the latest 2.1.x on the box.

[martincostello]

I thought the “don’t specify a version” feature was being deprecated after it caused different issues?

[skyflyer]

@natemcmaster, can you also please clarify the relation between target framework (moniker) e.g. netcoreapp2.1 and nuget packages, where runtime version is apparently also selected?

1) If we're not "pinning" the version numbers locally, does this mean that we're potentially running different frameworks on different machines? What kind of differences can we expect? (I know SCD works, because I've deployed 2.1.3 with SCD to Azure)

2) I was under the impression (and please, correct me, so I (we) can better understand and learn) that when I select a target framework (whether be it netcoreapp2.0, netcoreapp2.1, netstandard2.0, etc) I am limiting supported runtimes. Runtimes have patches, so runtimes 2.1.1, 2.1.2, 2.1.3... should all be compatible with the netcoreapp2.1 target framework, but the compiled code should run on all of them. I suppose I'm missing an important bit, where NuGet package references are not only package references, but actually runtimes as well? So, color me confused. The NuGet description for Microsoft.AspNetCore.App says:

Provides a default set of API for building an ASP.NET Core application, and also includes API for third-party integrations with ASP.NET Core. This package requires the ASP.NET Core runtime. This runtime is installed by the .NET Core SDK, or can be acquired separately using installers available at https://aka.ms/dotnet-download.

So it isn't a runtime, but it is a "meta" package, which depends on bunch of other packages. But my dotnet --info command shows those "names" as runtimes:

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.0-rc2-3002702 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

So I'm guessing, they are meta packages, which also require a runtime. Uff. :) The Target Frameworks page does not mention runtimes at all. Perhaps I got it all backwards, because it does not make much sense, now that I reread it... :(

3) Attempt 3:

Target framework: specify a set of APIs which should be available in the runtime - I hope this one makes sense. So, if my app targets netcoreapp2.1, I expect a certain set of APIs to be available.

Runtime: supports a given set of target frameworks, by providing required APIs.

NuGet package Microsoft.AspNetCore.All: no idea, if it is not just a meta package

Please, educate us. :)

[goodcoffeecode]

Hi all,

This has been stopping our CI/CD pipelines deploying to App Services since Friday :(. We've tried adding a .NET Core Tool Installer step to our builds and setting the SDK version to 2.1.300. This successfully runs dotnet restore from the correct SDK directory BUT it then restores 2.1.4 to every project. What are we doing wrong please?

Thanks,

Chris

[vijayrkn]

@goodcoffeecode -

Do you have a version specified for this package Microsoft.AspNetCore.App in the csproj?

<PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.3"/>

If so, can you remove the version from this and everything should work:

<PackageReference Include="Microsoft.AspNetCore.App" />

[goodcoffeecode]

We took the decision to pin versions so we knew what we were releasing. They currently on 2.0.1.

[vijayrkn]

This seems to be the latest version on App Service. Sdk: 2.1.302 Runtime: 2.1.2

setting the SDK version to 2.1.300. This successfully runs dotnet restore from the correct SDK directory BUT it then restores 2.1.4 to every project.

@natemcmaster - Do you know why updating the version to 2.1300 would force a restore of 2.1.4 version when the version is pinned at 2.0.1?

[natemcmaster]

By the way, I believe 2.1.3 has rolled out to West Central US now. Please let us know if apps running in this region started having issues.

Do you know why updating the version to 2.1.300 would force a restore of 2.1.4 version when the version is pinned at 2.0.1?

I would have to see repro steps (preferably a binary log file) and a project.assets.json file. You can gather binlogs by running dotnet restore -bl. The assets file is found in the obj/ folder of projects.

[natemcmaster]

For anyone looking for more information on Microsoft.AspNetCore.App, rollforward, target frameworks, etc., I recommend taking a look at this doc. https://docs.microsoft.com/en-us/aspnet/core/fundamentals/metapackage-app?view=aspnetcore-2.1

It was updated recently to include a section about patch updates.

[niemyjski]

@davidebbo I've been running into deployment issues (kudu related) and to top that off I updated the packages days after release of 2.1.3 and hit another deployment issue in a testing slot because this (2.1.3) wasn't deployed. This is starting to get to a level of unacceptable and really wasting a lot of peoples time with 502.5 errors. If a package is on nuget, it should work in azure 100% of the time. Or let us install an azure site extension that isn't 2.2 preview that is 100% in sync with latest deployed runtime (where I don't have to manually update it)

[natemcmaster]

@skyflyer I wrote up more details about Microsoft.AspNetCore.App and tried to explain why you're getting HTTP 502.5 errors. Let me know if this clarifies the relationship between shared frameworks, metapackages, and target frameworks. https://natemcmaster.com/blog/2018/08/29/netcore-primitives-2/

@niemyjski An important piece to understand is that Microsoft.AspNetCore.App is not a normal nuget package. Details in the docs link above and in my post. I recommend using the lowest possible version of the package in your project. When Azure rolls out 2.1.x patches, you apps automatically rollforward onto the newer version without you needing to redeploy.

cc @davidfowl @damianedwards

[rasmustherkelsen]

I am seeing weird issues where logic app deployments cannot see the Azure Functions they use during deployment from VSTS. My CD pipeline deploys Azure Functions that are then used by logic apps deployed via an ARM template. Those templates break randomly with bad request errors pointing to the functions endpoints. I can create a logic app manually, point it to the function and it works and the code-behind is the same (when viewed in automation script, urls and everything). Would that be related to this deployment of Azure Functions?

This behaviour started about 9 in the morning Danish time today - around midnight pacific time.

[davidebbo]

@rasmustherkelsen I think you're in the wrong place. This issue is to track exactly one thing: the deployment of .NET Core 2.1.3 to App Service. Please open different issues (e.g. in Functions repos) for other things. Thanks!

[rasmustherkelsen]

@davidebbo ups sorry. Opened on the MSDN forum. Feel free to delete my comment.

[skyflyer]

@natemcmaster, re your blog post: excellent, explains a lot. Thanks!

[waconrad]

Has .NET Core 2.1.3 been rolled out to any more Azure regions? More specifically to the US South Central or North Central regions.

[davidebbo]

@waconrad it's in North Central US. Please verify that you see it there.

[waconrad]

I confirmed that 2.1.3 is deployed to the US North Central region. Thanks!

[davidebbo]

It is now deployed everywhere.

[skyflyer]

@davidebbo, interestingly enough, what worked yesterday, failed today. I have an app deployed in West Europe and the app stopped responding today. The app is deployed using framework dependent deployment procedure and versions are not set explicitly in the config file. From the project file:

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.4.0" />
  </ItemGroup>

Now, the error on AppService is:

Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.Identity.Core, Version=2.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
   at Diary.Api.Startup.ConfigureServices(IServiceCollection services)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()

I tried fixing it with SCD deployment, but kudu failed (zip deploy):

``` 2018-08-30T09:19:13 Copying file: 'Diary.Api.pdb' 2018-08-30T09:19:13 Failed exitCode=1, command="kudusync" -v 50 -f "D:\local\Temp\zipdeploy\extracted" -t "D:\home\site\wwwroot" -n "D:\home\site\deployments\1af2feb183d74cdea3960a051335aee0\manifest" -p "D:\home\site\deployments\4d48e2c1181745b583f748ce40750bc7\manifest" -i ".git;.hg;.deployment;deploy.cmd" 2018-08-30T09:19:13 An error has occurred during web site deployment. 2018-08-30T09:19:13 Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Diary.Api.pdb" D:\Program Files (x86)\SiteExtensions\Kudu\77.10823.3533\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd" 2018-08-30T09:19:13 Error occurred, type: error, text: Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Diary.Api.pdb" D:\Program Files (x86)\SiteExtensions\Kudu\77.10823.3533\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd", stackTrace: at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 235 at Kudu.Core.Infrastructure.Executable.ExecuteWithProgressWriter(ILogger logger, ITracer tracer, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 137 at Kudu.Core.Deployment.Generator.ExternalCommandBuilder.RunCommand(DeploymentContext context, String command, Boolean ignoreManifest, String message) in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\Generator\ExternalCommandBuilder.cs:line 96 2018-08-30T09:19:13 2018-08-30T09:19:13 Error occurred, type: error, text: Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Diary.Api.pdb" D:\Program Files (x86)\SiteExtensions\Kudu\77.10823.3533\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd", stackTrace: at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 235 at Kudu.Core.Infrastructure.Executable.ExecuteWithProgressWriter(ILogger logger, ITracer tracer, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 137 at Kudu.Core.Deployment.Generator.ExternalCommandBuilder.RunCommand(DeploymentContext context, String command, Boolean ignoreManifest, String message) in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\Generator\ExternalCommandBuilder.cs:line 114 at Kudu.Core.Deployment.Generator.GeneratorSiteBuilder.Build(DeploymentContext context) in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\Generator\GeneratorSiteBuilder.cs:line 38 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Kudu.Core.Deployment.DeploymentManager.d__36.Mo ```

I then killed the processes and retried deployment, which fixed it.

Can you comment on that (on the version error, I understand what kudu error is about)? Are the environments in still flux? Even so, I wouldn't expect it to be half-deployed?

If I understand the situation correctly, the "rollforward", explained by @natemcmaster didn't work in this case, even though the versions were not pinned.

[josephayoung]

We had the same issue this morning in Central US -- the overnight upgrade brought down all of our .NET Core 2.1 apps with 502.5 errors. Specifically:

Application startup exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileLoadException: Could not load file or assembly 'Microsoft.AspNetCore.Connections.Abstractions, Version=2.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference.

Our .csproj files were set to 2.1.1:

  <ItemGroup>
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="5.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.1" />
  </ItemGroup>

The way we fixed it was to upgrade our hard-coded versions to 2.1.3 -- but this is obviously a horrible solution:

  <ItemGroup>
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="5.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.3" />
  </ItemGroup>

After the apps were back up and running, I tried to use the "version-agnostic" version of the PackageReference referenced above as a "permanent" fix to this issue, and it didn't work -- this took the apps back down with the same 502.5 error and the same log entry (unable to find 2.1.3 references):

  <ItemGroup>
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="5.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.App" />
  </ItemGroup>

So something seems off, and as @skyflyer mentioned, either the "rollforward" isn't working for whatever reason, or we don't fully understand the right combination of settings.

I know we could probably fix this permanently by switching to a self-contained publish, but I'd like to avoid that, as it renders the auto-managing of .NET Core updates on App Service totally useless, and increases the size of our publish by 10,000% or so.

[davidebbo]

@skyflyer @josephayoung Can you run dotnet --info from Kudu console, to make sure the new bits are there? Also, if you can share the name of an app in this state, that will help us investigate (you can potentially set up a repro in a new app). Thanks!

[josephayoung]

Here is our dotnet --info for one of the affected deployments:

.NET Core SDK (reflecting any global.json):
 Version:   2.1.401
 Commit:    91b1c13032

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.14393
 OS Platform: Windows
 RID:         win10-x86
 Base Path:   D:\Program Files (x86)\dotnet\sdk\2.1.401\

Host (useful for support):
  Version: 2.1.3
  Commit:  124038c13e

.NET Core SDKs installed:
  1.1.8 [D:\Program Files (x86)\dotnet\sdk]
  1.1.10 [D:\Program Files (x86)\dotnet\sdk]
  2.1.101 [D:\Program Files (x86)\dotnet\sdk]
  2.1.301 [D:\Program Files (x86)\dotnet\sdk]
  2.1.401 [D:\Program Files (x86)\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.10 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.1 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

App is "appyhealth-dev" in Central US -- it is up and running now with the hard-coded fix I mentioned above, but I can redeploy it with the previous codebase to try and "break" it again if you need.

[skyflyer]

@davidebbo, here is output from Kudu:

``` D:\home>dotnet --info .NET Core SDK (reflecting any global.json): Version: 2.1.401 Commit: 91b1c13032 Runtime Environment: OS Name: Windows OS Version: 10.0.14393 OS Platform: Windows RID: win10-x86 Base Path: D:\Program Files (x86)\dotnet\sdk\2.1.401\ Host (useful for support): Version: 2.1.3 Commit: 124038c13e .NET Core SDKs installed: 1.1.8 [D:\Program Files (x86)\dotnet\sdk] 1.1.10 [D:\Program Files (x86)\dotnet\sdk] 2.1.101 [D:\Program Files (x86)\dotnet\sdk] 2.1.300-rc1-008673 [D:\Program Files (x86)\dotnet\sdk] 2.1.302 [D:\Program Files (x86)\dotnet\sdk] 2.1.401 [D:\Program Files (x86)\dotnet\sdk] .NET Core runtimes installed: Microsoft.AspNetCore.All 2.1.0-rc1-final [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.2 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.All 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All] Microsoft.AspNetCore.App 2.1.0-rc1-final [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.2 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 1.0.10 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 1.1.7 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.0-rc1 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.2 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App] To install additional .NET Core runtimes or SDKs: https://aka.ms/dotnet-download ```

I'll can share with you the complete app or the name, but I'd rather not do it publicly. Please contact me at my dot at gmail. You can see it in my profile page. The app is running in West Europe.

[davidebbo]

@josephayoung so the update is there. Will probably need @natemcmaster's help to figure out what's going on. I assume you don't want your app to be down, so maybe it's best if you can set up a separate one with the repro?

[davidebbo]

@skyflyer sorry, I don't understand what "my dot at gmail" means, and I don't see it in your profile page. Can you clarify?

[josephayoung]

Yep, sure! Just deployed a repro project to https://appyhealth-app-service-issue-65.azurewebsites.net. All I did was remove the version number from the package reference, and the 502.5 error is back.

Relevant lines from the .csproj for this deployment:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <TypeScriptCompileBlocked>true</TypeScriptCompileBlocked>
    <TypeScriptToolsVersion>Latest</TypeScriptToolsVersion>
    <IsPackable>false</IsPackable>
    <SpaRoot>ClientApp\</SpaRoot>
    <DefaultItemExcludes>$(DefaultItemExcludes);$(SpaRoot)node_modules\**</DefaultItemExcludes>

    <!-- Set this to true if you enable server-side prerendering -->
    <BuildServerSideRenderer>false</BuildServerSideRenderer>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="5.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.AspNetCore.SignalR" Version="1.0.3" />
    <PackageReference Include="Microsoft.AspNetCore.SpaServices.Extensions" Version="2.1.1" />
    <PackageReference Include="Microsoft.Identity.Client" Version="1.1.4-preview0002" />
    <PackageReference Include="Stripe.net" Version="18.0.0" />
  </ItemGroup>

[davidebbo]

@josephayoung thanks, that should be all we need to start investigating. We'll follow up.

[natemcmaster]

If you are getting failures due to "System.IO.FileLoadException", you are running into https://github.com/dotnet/core-setup/issues/4376. This is a bug in rollforward I identified last month while preparing the 2.1.3 patches, but were not able to get fixed in time for the 2.1.3 release.

The fix is to update the project to <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.3" />.

The cause of this issue, as described in https://github.com/dotnet/core-setup/issues/4376, is unintentionally upgrading out of the shared framework. For example:

    <PackageReference Include="Microsoft.AspNetCore.App" /> <!--   The implicit version is 2.1.1 -->
    <PackageReference Include="Microsoft.AspNetCore.SignalR" Version="1.0.3" />

Microsoft.AspNetCore.App 2.1.1 depends on Microsoft.AspNetCore.SignalR 1.0.1, but your project has upgrade to Microsoft.AspNetCore.SignalR 1.0.3. When you dotnet publish, you get Microsoft.AspNetCore.SignalR 1.0.3 AND ⚠️ its transitive dependencies in your publish folder. This is the publish trimming pitfall I described in https://natemcmaster.com/blog/2018/08/29/netcore-primitives-2/.

This results in the following

AspNetCore.App	M.A.Connections.Abstractions (in shared framework)	M.A.Connections.Abstractions (in publish dir)	Which M.A.Connections.Abstractions should load
2.1.1	2.1.1	2.1.2	publish dir
2.1.3	2.1.3	2.1.2	shared framework*

* This is bug https://github.com/dotnet/core-setup/issues/4376. The framework version should load, but the host picked the publish dir version which causes a runtime downgrade. Hopefully we can get @steveharter's help in fixing this for 2.1.5.

cc @damianedwards @davidfowl

[skyflyer]

@davidebbo sorry, typo: my first name dot my last name @gmail.com

[davidebbo]

@skyflyer got a repro, and the issue has now been identified (and is known), per @natemcmaster's response (it is not specific to App Service).

[josephayoung]

OK, thanks for the update! One clarification, though -- the initial broken app was still referencing Microsoft.AspNetCore.SignalR 1.0.1 -- I manually updated this afterwards to 1.0.3 to match the Core 2.1.3 dependency. So I'm not sure that that particular package was involved in the issue. @natemcmaster

[josephayoung]

Disregard my previous comment -- SignalR was actually on 1.0.2 originally, so since that is still higher than the .App 2.1.1 dependency, that must have triggered the publish trimming issue once the Azure upgrade rolled out.

The .csproj at the time of the original 502.5 error that popped up overnight was this:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <TypeScriptCompileBlocked>true</TypeScriptCompileBlocked>
    <TypeScriptToolsVersion>Latest</TypeScriptToolsVersion>
    <IsPackable>false</IsPackable>
    <SpaRoot>ClientApp\</SpaRoot>
    <DefaultItemExcludes>$(DefaultItemExcludes);$(SpaRoot)node_modules\**</DefaultItemExcludes>

    <!-- Set this to true if you enable server-side prerendering -->
    <BuildServerSideRenderer>false</BuildServerSideRenderer>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AutoMapper.Extensions.Microsoft.DependencyInjection" Version="5.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.App" Version="2.1.1" />
    <PackageReference Include="Microsoft.AspNetCore.SignalR" Version="1.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.SpaServices.Extensions" Version="2.1.1" />
    <PackageReference Include="Microsoft.Identity.Client" Version="1.1.4-preview0002" />
    <PackageReference Include="Stripe.net" Version="18.0.0" />
  </ItemGroup>

[skyflyer]

I've been thinking that the problem might be affecting me because I'm having two apps under the same app service plan and one of them has been using SCD (because of the errors encountered). I redeployed both apps using framework deployment procedure and to make matters more interesting, one of them works and one of them fails:

2018-08-31T05:42:51 env XPROC_TYPENAME=Microsoft.Web.Hosting.Transformers.ApplicationHost.SiteExtensionHelper, Microsoft.Web.Hosting, Version=7.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
2018-08-31T05:42:51 env XPROC_METHODNAME=Transform
2018-08-31T05:42:51 Start 'Microsoft.AspNetCore.AzureAppServices.SiteExtension' site extension transform
2018-08-31T05:42:51 StartSection Executing InsertIfMissing (transform line 7, 23)
2018-08-31T05:42:51 on /configuration/system.webServer
2018-08-31T05:42:51 Applying to 'configuration' element (no source line info)
2018-08-31T05:42:51 EndSection Done executing InsertIfMissing
2018-08-31T05:42:51 StartSection Executing InsertIfMissing (transform line 8, 16)
2018-08-31T05:42:51 on /configuration/system.webServer/runtime
2018-08-31T05:42:51 Applying to 'system.webServer' element (no source line info)
2018-08-31T05:42:51 Inserted 'runtime' element
2018-08-31T05:42:51 EndSection Done executing InsertIfMissing
2018-08-31T05:42:51 StartSection Executing InsertIfMissing (transform line 9, 31)
2018-08-31T05:42:51 on /configuration/system.webServer/runtime/environmentVariables
2018-08-31T05:42:51 Applying to 'runtime' element (no source line info)
2018-08-31T05:42:51 EndSection Done executing InsertIfMissing
2018-08-31T05:42:51 StartSection Executing InsertOrAppendAttribute (transform line 10, 315)
2018-08-31T05:42:51 on /configuration/system.webServer/runtime/environmentVariables/add[@name='DOTNET_ADDITIONAL_DEPS']
2018-08-31T05:42:51 Applying to 'environmentVariables' element (no source line info)
2018-08-31T05:42:51 EndSection Done executing InsertOrAppendAttribute
2018-08-31T05:42:51 StartSection Executing InsertOrAppendAttribute (transform line 11, 158)
2018-08-31T05:42:51 on /configuration/system.webServer/runtime/environmentVariables/add[@name='DOTNET_SHARED_STORE']
2018-08-31T05:42:51 Applying to 'environmentVariables' element (no source line info)
2018-08-31T05:42:51 EndSection Done executing InsertOrAppendAttribute
2018-08-31T05:42:51 StartSection Executing InsertOrAppendAttribute (transform line 12, 146)
2018-08-31T05:42:51 on /configuration/system.webServer/runtime/environmentVariables/add[@name='ASPNETCORE_HOSTINGSTARTUPASSEMBLIES']
2018-08-31T05:42:51 Applying to 'environmentVariables' element (no source line info)
2018-08-31T05:42:51 EndSection Done executing InsertOrAppendAttribute
2018-08-31T05:42:51 Successful 'D:\home\SiteExtensions\Microsoft.AspNetCore.AzureAppServices.SiteExtension\applicationHost.xdt' site extension transform
2018-08-31T05:42:51 sandboxproc.exe complete successfully. Elapsed = 475.00 ms
2018-08-31 05:42:53.094 +00:00 [Critical] Microsoft.AspNetCore.Hosting.Internal.WebHost: Application startup exception
System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Extensions.Identity.Core, Version=2.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'Microsoft.Extensions.Identity.Core, Version=2.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Diary.Api.Startup.ConfigureServices(IServiceCollection services)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.Initialize()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()

I even killed the processes before trying to run it. Both apps use ASP.NET Core Identity. The package references are very small:

Complete csproj

``` netcoreapp2.1 13422dd2-c8c9-4397-86c5-62ca5b5bf779 ```

Both apps reference the shared "Data" project, which has the following references:

Data csproj

``` netcoreapp2.1 ```

The runtimeconfig.json files are actually generated as 2.1.1 (?):

Runtime configs

``` $ cat ./Diary.Backend/dist/Diary.Backend.runtimeconfig.json { "runtimeOptions": { "tfm": "netcoreapp2.1", "framework": { "name": "Microsoft.AspNetCore.App", "version": "2.1.1" }, "configProperties": { "System.GC.Server": true } } } $ cat ./Diary.Api/dist/Diary.Api.runtimeconfig.json { "runtimeOptions": { "tfm": "netcoreapp2.1", "framework": { "name": "Microsoft.AspNetCore.App", "version": "2.1.1" }, "configProperties": { "System.GC.Server": true } } } ```

Kudu is giving me error on deploys (not all of them) even if the processes are stopped / killed. Don't know which process might be holding on to that .pdb file. (file handle search did not find it)

``` 2018-08-31T05:55:24 Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Diary.Api.pdb" D:\Program Files (x86)\SiteExtensions\Kudu\77.10830.3542\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd" 2018-08-31T05:55:24 Error occurred, type: error, text: Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Diary.Api.pdb" D:\Program Files (x86)\SiteExtensions\Kudu\77.10830.3542\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd", stackTrace: at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 235 at Kudu.Core.Infrastructure.Executable.ExecuteWithProgressWriter(ILogger logger, ITracer tracer, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 137 at Kudu.Core.Deployment.Generator.ExternalCommandBuilder.RunCommand(DeploymentContext context, String command, Boolean ignoreManifest, String message) in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\Generator\ExternalCommandBuilder.cs:line 96 2018-08-31T05:55:24 2018-08-31T05:55:24 Error occurred, type: error, text: Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Diary.Api.pdb" D:\Program Files (x86)\SiteExtensions\Kudu\77.10830.3542\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd", stackTrace: at Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer, Func`2 onWriteOutput, Func`2 onWriteError, Encoding encoding, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 235 at Kudu.Core.Infrastructure.Executable.ExecuteWithProgressWriter(ILogger logger, ITracer tracer, String arguments, Object[] args) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\Executable.cs:line 137 at Kudu.Core.Deployment.Generator.ExternalCommandBuilder.RunCommand(DeploymentContext context, String command, Boolean ignoreManifest, String message) in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\Generator\ExternalCommandBuilder.cs:line 114 at Kudu.Core.Deployment.Generator.GeneratorSiteBuilder.Build(DeploymentContext context) in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\Generator\GeneratorSiteBuilder.cs:line 38 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Kudu.Core.Deployment.DeploymentManager.d__36.MoveNext() in C:\Kudu Files\Private\src\master\Kudu.Core\Deployment\DeploymentManager.cs:line 661 ```

---

How I eventually fixed it (not sure which of the changes fixed it):

• I modified the Data project to use the 2.1.3 reference of Microsoft.AspNetCore.Identity.EntityFrameworkCore
• I removed the contents of wwwroot from the debug console (that fixed the kudu deployment errors)

The app went live. :) I have the failing setup in the source control repo should you want to inspect it.

We have two apps, one which was working (same dependencies) and one which was not. Same app service plan. The apps stopped working when the rollover to 2.1.3 started, though I have to add, that the app which is and was working, was using SCD at the time.

[goodcoffeecode]

@vijayrkn Our project is a bit of a mix at the moment. The target framework is 462, but it contains some .Net Core 2x code too. Microsoft.AspNetCore.App is not compatible with net462, so we can't include it.

[goodcoffeecode]

OK, so we're still broken here :(. We've tested a number of combinations of agent pools, SDK versions and deployment methods. Before the upgrade, we had been successfully deploying this particular App Service for years. Here are our results:

Method	SDK	Runtime	Agent Pool	Result
VSDeploy	2.1.401	2.1.3		502
VSDeploy	2.1.302	2.1.3		Success
VSTS	2.1.401	2.1.3	Hosted VS2017	502
VSTS	2.1.302	2.1.3	Hosted VS2017	502
VSTS	2.1.302	2.1.3	Hosted	Success
VSTS	2.1.401	2.1.3	Hosted	Build failed "File type not recognised"

Agent Pools:

Capability	Hosted	Hosted VS2017
Agent.Version	2.139.1	2.139.1
AzurePS	1.0.0	3.6.0
DotNetFramework	4.0.30319	4.0.30319
MSBuild	14	15
VisualStudio	14	2017

How has a minor revision increment caused so many errors? This feels like it should be v3.0.0, not 2.1.4.

[davidebbo]

@goodcoffeecode let's focus on issues that relate to running Core in App Service. If you get errors building in VSTS, please open an issue on https://github.com/Microsoft/vsts-tasks.

When you get 502, please follow this guidance to figure out what the error is. Hopefully that will give enough hints to figure things out.

[natemcmaster]

@skyflyer your app was broken for the reason I explained above in https://github.com/Azure/app-service-announcements-discussions/issues/65#issuecomment-417379227. The package reference to Microsoft.AspNetCore.Identity.EntityFrameworkCore in Data.csproj causes issues with https://github.com/dotnet/core-setup/issues/4376. The app-local version overrides the shared framework, causing issues when your code runs other libraries, such as Microsoft.AspNetCore.Identity.UI, which depend on Microsoft.Extensions.Identity.Core >= 2.1.3.0. I was able to reproduce the problem locally. I used your project setup to verify a fix we are planning for 2.1.5.

[skyflyer]

@natemcmaster, great! I hope we don't the same errors during the next upgrade :)

[niemyjski]

@davidebbo What's the status of 2.1.4?

[davidebbo]

@niemyjski sorry, we missed the announcement on this one, but it got deployed within a couple days of release. So it's there!