Functions no longer running in Azure

dsm0880 commented 6 years ago

When publishing to a Function app I can no longer run my functions. If I completely delete the function app and recreate it, provision a simple HTTP Trigger template everything works. After I publish I receive the following error when accessing an individual function:

The function runtime is unable to start. System.Private.CoreLib: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621). System.Private.CoreLib: Could not load file or assembly 'Microsoft.AspNetCore.Mvc.Abstractions, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Session Id: d1a1852c325d47cd9e76a32ddc097293

Timestamp: 2018-02-22T21:38:33.235Z

I also notice I cannot access the host keys when going to the root function app. I get the following error:

Service Unavailable - https://c365-funcitons2.azurewebsites.net/admin/host/keys Session Id: d1a1852c325d47cd9e76a32ddc097293

Timestamp: 2018-02-22T21:42:10.525Z

Investigative information

Please provide the following:

Timestamp: 2018-02-22T21:38:33.235Z
Function App version: 2.0-beta
Function App name: ---
Function name(s): Orchestration
Invocation ID: d1a1852c325d47cd9e76a32ddc097293
Region: East US

Repro steps

Provide the steps required to reproduce the problem:

Create a new Function App
Set it to beta
Create a function using a template, things will work fine
Create a Durable Function locally and then publish. At least for myself this is no longer working. It was working earlier today.

Known workarounds

I don't have any workarounds and I consider this a pretty bad issue for myself. I can tell you a related ticket had the same issue where the runtime had a similar error regarding runtime not being able to find the MVC.Abstractions.dll. Since it was local I could manually replace the dll, but of course I cannot do this in azure.

tsmccartan commented 6 years ago

I have the same issue here. Function App worked fine with ~1, but when I switched to beta, it gives me the error Service Unavailable and I can not call any of the Functions anymore. I was trying out the Function App Chaining using Durable Functions.

dsm0880 commented 6 years ago

@tsmccartan Haven't received any responses yet, but we are still investigating on our end. It was super weird basically someone would add a single function and it would break everything by getting the message I posted above. Are you getting the same error when you go to a specific function? I got the "Service Unavailable" if I went to the function app url. If I reverted back and removed the newly added function everything would work as expected.

On a somewhat related note if you are having any trouble we got remote debugging to work by attaching to dotnet.exe instead of the w3wp.exe which everyone says to do.

tsmccartan commented 6 years ago

@dsm0880 It generally tells me that the Service in unavailable, when I go to a specific Function I get the following (Which makes me think that maybe the published functions are not compatible with the assemblies for 2.x?). I did not redeploy any of the functions. I have several Azure Functions that use KeyVault, ServiceBus, etc. I was testing Durable Functions and added the NuGet Package and the sample functions, so I can chain together some Functions. I found out that when I tested it, they did not work. I looked at the docs, and it said the Function Apps had to be configured for 2.0 (Beta), so I switched it, and after that none of the functions worked anymore? I have restarted, stopped, started, etc.

Directions Followed to Target: https://docs.microsoft.com/en-us/azure/azure-functions/set-runtime-version

ERROR The function runtime is unable to start. System.Private.CoreLib: Exception has been thrown by the target of an invocation. Microsoft.ServiceBus: The type initializer for 'Microsoft.ServiceBus.Messaging.Constants' threw an exception. Microsoft.ServiceBus: Could not load file or assembly 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The system cannot find the file specified.

dsm0880 commented 6 years ago

@tsmccartan You may have a different issue, assuming everything compiles fine locally? Ensure you are actually running with V2 of core tools. If you go to %localappdata% in file explorer do you see Azure.Functions.V2.Cli? If not make sure you have installed the latest azure functions extension in VS Extensions and Updates. After that make sure you have <AzureFunctionsVersion>v2</AzureFunctionsVersion> in the csproj file. Then recompile and see if you get any errors.

tsmccartan commented 6 years ago

@dsm0880 Thanks for the information. I definitely did not do that, as my project still targets net461 I think I will just wait till this feature becomes more prime time. Seems like a bunch of work to get the project up and running with dotnetcore2, publish it and get it to work with Azure Function Beta....

Thanks for all your quick feedback, sorry my issue is not exactly like yours and won't be able to help with yours.

audipen commented 6 years ago

@dsm0880 I do get the same error and have the latest Azure Functions extension installed in VS and the csproj file also looks fine. The error is

The function runtime is unable to start. System.Private.CoreLib: Exception has been thrown by the target of an invocation. Microsoft.ServiceBus: The type initializer for 'Microsoft.ServiceBus.Messaging.Constants' threw an exception. Microsoft.ServiceBus: Could not load file or assembly 'System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The system cannot find the file specified. Session Id: 3e83a74f5a254b328ff24110b66f2096

Timestamp: 2018-03-14T15:58:11.358Z

mathewc commented 6 years ago

We did a new runtime update this week that addresses some of the issues above, however the we're still updating Durable Functions preview to work with that new release. @ConnorMcMahon can provide an update for that.

oluatte commented 6 years ago

@davidebbo

As I mentioned in the other thread, we're having this issue at the top of this thread with the new runtime and it's blocking us.

As requested, here are the function details

East Region 2018-03-30T00:17:36.960 Function started (Id=13990dee-f7fe-42a8-ab5e-a55448714ac8)

davidebbo commented 6 years ago

@mayoatbm Some questions:

Did changing the App Setting get it back to working?
Are the files right now in a state such that if you switched back to latest Beta build, it would fail in the same way? Or did you make change so the file that make it behave differently?

oluatte commented 6 years ago

@davidebbo

Yes, it did
The files in the function app i shared above are pre - your upgrade instructions for the new runtime. That function app uses sdk 1.0.7 and beta4 versions of the nuget packages.

We have six function apps impacted by this issue. I tried upgrading one of them per your instructions and i got the error at the top of this thread. Unfortunately i can't share the details because I can't it to run to capture that information ((host is not running). I can send it to you privately if you'd like.

davidebbo commented 6 years ago

@mayoatbm looking at our logs, I see a very different error in Azure:

System.MissingMethodException : Method not found: 'Void Microsoft.Azure.WebJobs.Host.Config.FluentBindingRule`1.BindToInput(System.Func`2<!0,!!0>)'.
   at BNSF.UAS.Platform.IoTHubManager.DI.DependencyConfiguration.Initialize(ExtensionConfigContext context)
   at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.InvokeExtensionConfigProviders(ExtensionConfigContext context) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\JobHostConfigurationExtensions.cs : 421
   at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.CreateStaticServices(JobHostConfiguration config) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\JobHostConfigurationExtensions.cs : 112
   at Microsoft.Azure.WebJobs.JobHost.InitializeServices() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\JobHost.cs : 470
   at Microsoft.Azure.WebJobs.JobHost.get_Services() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\JobHost.cs : 482
   at Microsoft.Azure.WebJobs.Script.Utility.CreateMetadataProvider(JobHost host) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Utility.cs : 372
   at Microsoft.Azure.WebJobs.Script.ScriptHost.LoadBindingExtensions(IEnumerable`1 functionMetadata,JObject hostConfigObject) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 849
   at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize() at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 303
   at Microsoft.Azure.WebJobs.Script.ScriptHostManager.RunAndBlock(CancellationToken cancellationToken) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHostManager.cs : 194

This suggests that you have your own custom extension that was built against the previous runtime, so it's hitting a method that's now gone (or changed). You will need to rebuild your extension against the latest packages.

oluatte commented 6 years ago

@davidebbo

That's the error we get with the new runtime using our old code (make sense). The problem is that upgrading gives us the abstractions error. There's another function app in that same resource group that has the new nuget packages and sdk per your instructions. That gives the error at the top of this thread.

Does that clarify things?

oluatte commented 6 years ago

@davidebbo This issue might also be related.

davidebbo commented 6 years ago

@mayoatbm yes, that helps. So to summarize:

it became broken in Azure with the MissingMethodException
you tried to upgrade things locally in VS, but then trying to run it in VS resulted in the Microsoft.AspNetCore.Mvc.Abstractions error.

Problem may be that VS still needs to take an update. For now, stay with old build, and we'll figure this out.

/cc @fabiocav

oluatte commented 6 years ago

@davidebbo

Great.

Yes, that's a good summary. The only clarification I will make is that we get this "abstractions" error in Visual Studio and in the Core Runtime from npm, and in the cloud. So it's everywhere.

Can you guarantee that the old version of the runtime will be available until this issue is resolved?

Also, what is the likelihood that this situation (deploying a runtime with known breaking changes) happens again? We know that functions v2 is in beta but sudden runtime failures in Azure are a whole different ballgame than the other (nuget version dependencies, sdk vs cli, vs visual studio extension, function generation etc) issues we've seen.

Thank you for your responsiveness.

davidebbo commented 6 years ago

@mayoatbm can you try the following:

locally, from the root of your app, run dotnet publish
find the built artifacts under bin\Debug\netstandard2.0
you can test them locally using func host start (get the latest from npm, e.g. npm i -g azure-functions-core-tools@core)
deploy the bits to Azure.

davidebbo commented 6 years ago

@mayoatbm Just saw your other reply

Definitely, we will keep the old one until the dust settles and we're confident that everyone can move forward
this is a rough upgrade because there were breaking changes. And frankly we didn't do a very good job anticipating all the consequences.
Yes, it's possible that there will be other breaking changes, but we'll do a better job. e.g. we could have told users ahead of time that they could target the old version, and then move to the new one at their own pace

oluatte commented 6 years ago

@davidebbo

The new cli works. Thank you very much, that was fast. Unfortunately the function still doesn't work in azure (i deployed the output of dotnet publish to via the kudu console). I still get the abstractions exception there.

Also. Thank you for the assurances re: keeping the old runtime around and future upgrades. That helps with the level of concern going forward.

I also appreciate the fast, patient responses. Please keep us posted in this thread on when the azure part is fixed so we can perform the upgrade. Let me know if i can provide any additional information.

davidebbo commented 6 years ago

@mayoatbm is that still in the same app as I was looking at yesterday, or a different one? I do not see any updates on that one.

oluatte commented 6 years ago

It's the other one in that same resource group. Remember I couldn't get you the function details because the host isn't running.

davidebbo commented 6 years ago

@mayoatbm got it. I just took a look, but I'm seeing yet a different exception on that one:

System.ArgumentNullException : Value cannot be null.
Parameter name: value
   at Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.set_InstrumentationKey(String value)
   at BNSF.UAS.Platform.DataFabric.DI.DependencyConfiguration.SetupApplicationInsights() at C:\Projects\BNSF\Code\BMRepo\UAV Analytics MVP\bnsf-uas-platform\DataFabric\DataFabric\DI\DependencyConfiguration.cs : 116
   at BNSF.UAS.Platform.DataFabric.DI.DependencyConfiguration.SetupLogging(ExtensionConfigContext context,ContainerBuilder builder) at C:\Projects\BNSF\Code\BMRepo\UAV Analytics MVP\bnsf-uas-platform\DataFabric\DataFabric\DI\DependencyConfiguration.cs : 100
   at BNSF.UAS.Platform.DataFabric.DI.DependencyConfiguration.RegisterServices(ExtensionConfigContext context) at C:\Projects\BNSF\Code\BMRepo\UAV Analytics MVP\bnsf-uas-platform\DataFabric\DataFabric\DI\DependencyConfiguration.cs : 71
   at BNSF.UAS.Platform.DataFabric.DI.DependencyConfiguration.Startup(ExtensionConfigContext context) at C:\Projects\BNSF\Code\BMRepo\UAV Analytics MVP\bnsf-uas-platform\DataFabric\DataFabric\DI\DependencyConfiguration.cs : 43
   at BNSF.UAS.Platform.DataFabric.DI.DependencyConfiguration.Initialize(ExtensionConfigContext context) at C:\Projects\BNSF\Code\BMRepo\UAV Analytics MVP\bnsf-uas-platform\DataFabric\DataFabric\DI\DependencyConfiguration.cs : 34
   at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.InvokeExtensionConfigProviders(ExtensionConfigContext context) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\JobHostConfigurationExtensions.cs : 421
   at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.CreateStaticServices(JobHostConfiguration config) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\JobHostConfigurationExtensions.cs : 112
   at Microsoft.Azure.WebJobs.JobHost.InitializeServices() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\JobHost.cs : 470
   at Microsoft.Azure.WebJobs.JobHost.get_Services() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\JobHost.cs : 482
   at Microsoft.Azure.WebJobs.Script.Utility.CreateMetadataProvider(JobHost host) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Utility.cs : 372
   at Microsoft.Azure.WebJobs.Script.ScriptHost.LoadBindingExtensions(IEnumerable`1 functionMetadata,JObject hostConfigObject) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 849
   at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize() at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 303
   at Microsoft.Azure.WebJobs.Script.ScriptHostManager.RunAndBlock(CancellationToken cancellationToken) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHostManager.cs : 194

oluatte commented 6 years ago

@davidebbohttps://github.com/davidebbo

That's my fault. The app insights key was missing (I didn't copy over the app setting when I deployed via kudu). I just put it in there and thing seem to be working, which is great. I must have misread the timestamp on the error below (from our app insights) after I deployed it. Apologies for the subsequent confusion.

So, is this where we stand? Using the CLI to run locally works and dotnet publish with a manual deploy to azure works. But anything through Visual studio won't work until the extension is updated?

Thanks!

davidebbo commented 6 years ago

@mayoatbm yes, that's exactly where things stand. The VS extension update is hopefully the last step. @soninaren should be able to provide update on the progress.

oluatte commented 6 years ago

@davidebbo

Great. We will keep an eye out for it. Thanks again for all your help, we're in a much better than we were yesterday.

tsmccartan commented 6 years ago

@davidebbo I had a successfully running Durable Function as well as other functions running in Azure that was processing data. but these are now all failing in Azure with the following error: (I Get this error whenever I click on the function in the Azure Portal. When I run the function from Postman I get "Function host is not running")

Error: The function runtime is unable to start. Microsoft.Azure.WebJobs.Extensions.DurableTask: Method not found: 'Microsoft.Azure.WebJobs.Host.TraceWriter Microsoft.Azure.WebJobs.Host.Config.ExtensionConfigContext.get_Trace()'.

This is using the Beta version and DotNetCore. I didn't change any of my code that was running on a 5 minute timer function, but found out today that it has been failing every time now? Is this a known issue (As you stated above, there are breaking changes and it is NOT working with Durable Functions)? Is there anything I can do, to get this back processing? Or do I just need to wait for some update?

Thanks!

davidebbo commented 6 years ago

@tsmccartan please see https://github.com/Azure/app-service-announcements/issues/94.

tsmccartan commented 6 years ago

@davidebbo Thanks, this solved my issues and my functions are back operational, processing data! Do you know if this addresses the issues with creating Custom Telemetry in Applications Insights?

From this Post: https://github.com/Azure/azure-functions-host/issues/2473#issuecomment-376984633

UPDATE - I tried to add in the following code again, and it still fails with the following error.

EXCEPTION Could not load file or assembly 'Microsoft.ApplicationInsights, Version=2.5.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621) Could not load file or assembly 'Microsoft.ApplicationInsights, Version=2.5.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

I have the following NuGet Packages Referenced:

<PackageReference Include="Microsoft.ApplicationInsights" Version="2.5.1" />
<PackageReference Include="Microsoft.ApplicationInsights.PerfCounterCollector" Version="2.5.1" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="3.0.0-beta5" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="1.2.2-beta3" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Http" Version="3.0.0-beta5" />
<PackageReference Include="Microsoft.Extensions.Configuration.AzureKeyVault" Version="2.0.0" />
<PackageReference Include="System.Data.SqlClient" Version="4.4.0" />

davidebbo commented 6 years ago

It took a little longer than we would have wanted, but the updated Azure Functions and WebJobs Tools VS extension is now up. Please go ahead and upgrade. The new version is 15.0.40405.0.

Thanks for your patience on this.

ddobric commented 6 years ago

Had same issue after deploying .NET Azure Function in "Azure Function App", which was previously hosting .NET Core Azure Function. I have created a new Azure Function App and deployed .NET Azure Function. All worked fine.

Please note, I have checked "delete files" before deployment. It didn't help.

Azure / azure-functions-host