search

Azure / Azure-Functions

1.12k stars 198 forks source link

Internal Server Error - https://<app_name>.azurewebsites.net/admin/host/keys #719

Open btardif opened 6 years ago

btardif commented 6 years ago

From @handeeadiguzel on February 12, 2018 15:1

Please provide us with the following information:

OS and Browser

Windows & Chrome

Repro steps

  1. Create function app from azure portal
  2. Change runtime version to beta (2.0.11415.0 (beta))
  3. Enable deployment slots

Host keys are gone (portal doesn't show them anymore). And cannot add new keys.

Any error messages

Internal Server Error - https:///admin/host/keys Session Id: 40a3ea0af19f42ceae950c973d66b770

Timestamp: 2018-02-12T14:42:19.631Z

Mention any other details that might be useful.

Copied from original issue: Azure/azure-functions-ux#2334

btardif commented 6 years ago

@ahmelsayed can you take a look?

Neonkiller commented 6 years ago

Hello I'm struggling the same issue. But runtime version is 1. It just stopped working. Any updates on it?

APErebus commented 6 years ago

I'd like to get some more info on this as well. We are getting this after creating a Azure Function via Terraform & deploying via Octopus Deploy.

This was working perfectly fine a week ago, now it's just failing continually

worldspawn commented 6 years ago

I'm now getting this as well on a function after deploying a new release. :'(

no keys image

Error:

Internal Server Error - https://<name>.azurewebsites.net/admin/host/keys
Session Id: e2207e819afe493687b65cf4b1441f71

Timestamp: 2018-06-26T13:21:21.196Z
Error:

Internal Server Error - https://<name>.azurewebsites.net/admin/host/systemkeys/_master
Session Id: e2207e819afe493687b65cf4b1441f71

Timestamp: 2018-06-26T13:23:12.606Z

I do have a host.json in the file system. Its full of encrypted keys.

> ls
D:\home\data\Functions\secrets
host.json
btardif commented 6 years ago

@paulbatum can you triage this item?

worldspawn commented 6 years ago

@btardif We have identified a possible cause for this. We were creating our function app using the terraform azure provider which seems to also create a few basic settings. Two of those settings are: WEBSITE_CONTENTSHARE WEBSITE_CONTENTAZUREFILECONNECTIONSTRING

These settings arn't part of our deployment so they are being removed when we deploy. The act of removing them seems to break the function app. I'm told these settings are intended for consumption plan backed functions only.

We couldn't hack terraform into not doing that so we've just added them to our set of dev settings until the provider can be patched.

Gowiem commented 6 years ago

Any final resolution on this issue? I'm running into this when deploying a fresh Function App with a FUNCTIONS_EXTENSION_VERSION: beta AppSetting.

btardif commented 6 years ago

@ahmelsayed @paulbatum can you guys comment on this?

Davy-F commented 6 years ago

I'm also getting '500 Internal Server Error' when deploying a new, v2 C# Function App. I've even tried doing a simple Hello World app and still get the same behaviour.

Any updates on this?

SteveVoros commented 6 years ago

I had the same problem. It was caused because I applied additional App Settings configurations in a second step using PowerShell where I accidentally wiped out default App Settings. After that, redeployment of the Function App failed and the only fix was to delete and recreate the Function App.

Gowiem commented 6 years ago

@btardif Continuing to run into this issue with a function app deployed with FUNCTIONS_EXTENSION_VERSION: 2.0.11961-alpha.

Gowiem commented 6 years ago

For anyone running into this issue with a beta runtime version -- An important note that I have not found elsewhere - If you're adding an Application Insights Extension to your function which runs a 2.x Runtime then it will fail with this error. The fix is to include the AI App Setting key manually, remove the AI Extension from the Function App, and then restart. Then it will work.

kazinad commented 6 years ago

It seems, host keys aren't working in this build. I vote for that this is a bug.

pragathikadaganchi commented 6 years ago

I'm facing the same issue after being delpoyed new function app to azure with java runtime

Error:

Internal Server Error - https://scanandgo-warmup.azurewebsites.net/admin/host/keys Session Id: ae893841c0764d98bc44b939653295c5

Timestamp: 2018-09-09T03:03:00.461Z

xtellurian commented 6 years ago

Facing the same issue.

Steps to repro:

  1. Create new Functions App on Windows with App Service Plan.
  2. Change runtime to V2
  3. Create a new JS HTTP trigger function.

Expected result: Things are OK.

Actual Result: Host Keys are deleted and can't be created.

Error:

Internal Server Error - https://locafi-ingress-2.azurewebsites.net/admin/host/systemkeys/_master

Session Id: 9cb3c262b262401096ad8aaebf637e03

Timestamp: 2018-09-12T05:02:45.815Z
paulbatum commented 6 years ago

I am not able to reproduce, in general. I investigated a few cases where the function app name and timestamp were shared.

@xtellurian Your function app is hitting outbound connection exhaustion errors. It looks like the connections are being exhausted by other applications running on your same app service plan. I am not able to tell which application is doing it, but I can see there are a huge amount of connections (~1900) to a specific IP address and port. If you need additional details and you're OK with me sharing the full IP and port, let me know and I can do so.

@pragathikadaganchi I'm seeing a similar case to the one above - errors relating to outbound connection exhaustion. In your case I see ~500 connections to a specific IP and port. Again, if you're OK with me sharing full details, let me know and I will do so.

I have filed https://github.com/Azure/azure-functions-host/issues/3444 to track some improvements that can be made.

Also, it looks like there are additional logs that are missing in version 12050 that would be helpful in narrowing down further why exactly these connection issues are impacting the master key API. We have fixed the issue that is preventing these logs from being present. So once the new version of functions is deployed, we can take another look at the logs for apps that are still hitting this error, and they should help further troubleshooting.

xtellurian commented 6 years ago

@paulbatum Ah, OK that's a bug, I believe it's from not re-using the connection to IoT Hub (the function is acting as a proxy for HTTP messages into IoT Hub).

I'll try to repro in a new App Service

chris-rogala commented 6 years ago

Any update on this? I just ran into the same error today.

paulbatum commented 6 years ago

This investigation has stalled. Cases I looked at were due to other problems (see my recent reply above). If you post detailed information about your particular case we can take a look at our logs.

scottalankrieger commented 5 years ago

I'm running into this with a new Azure function app.

Internal Server Error - https://api-inscape.azurewebsites.net/admin/host/keys Session Id: 36e8a8ca2d4146eebabe5dc3aaad58d7

Timestamp: 2018-11-09T15:38:25.898Z

paulbatum commented 5 years ago

@ScottKriegerPD I looked at your case. The issue appears to be related to a missing storage account connection string:

System.InvalidOperationException : Secret initialization from Blob storage failed due to a missing Azure Storage connection string. If you intend to use files for secrets, add an App Setting key 'AzureWebJobsSecretStorageType' with value 'Files'.
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultSecretManagerProvider.CreateSecretsRepository() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\DefaultSecretManagerProvider.cs : 60
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultSecretManagerProvider.Create() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\DefaultSecretManagerProvider.cs : 48

You should see the same error if your look at your own logs in app insights. Functions expects to have a storage connection string named "AzureWebJobsStorage" and it uses this to manage secrets.

scottalankrieger commented 5 years ago

Thanks @paulbatum. I recreated the app and it's functioning now. I'll watch for that in the future.

sprabhus commented 5 years ago

I am too running into same issue with new the azure function app where I tried all the observations posted above.

Error: Internal Server Error - https://oneprofilesyncappdev.azurewebsites.net/admin/host/systemkeys/_master Session Id: a76915240e404f67b58729ae68be9c6a

Timestamp: 2018-11-28T09:12:45.049Z

paulbatum commented 5 years ago

@sprabhus I looked at your case, I see an error I have not seen before:

Microsoft.WindowsAzure.Storage.StorageException : One of the request inputs is out of range.
   at async Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd,IRetryPolicy policy,OperationContext operationContext,CancellationToken token)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.CreateIfNotExistsAsync(??)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository..ctor(String secretSentinelDirectoryPath,String accountConnectionString,String siteSlotName) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs : 65
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultSecretManagerProvider.CreateSecretsRepository() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\DefaultSecretManagerProvider.cs : 66
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultSecretManagerProvider.Create() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\DefaultSecretManagerProvider.cs : 48

@alrod any ideas about what might cause this?

alrod commented 5 years ago

@sprabhus can you please check "AzureWebJobsStorage" app setting. Can you connect to the storage using "Azure Storage Explorer"?

sprabhus commented 5 years ago

Yes AzureWebJobsStorage has the connection string to the storage account name-opSyncAppStorageDEV and i am able to connect to the storage account using storage explorer using the connection string in the AzureWebJobsStorage app setting. Please let me know if you need any further details.

MysticWolf commented 5 years ago

Hello, I got the same issue with an azure function created from Visual Code. My function was using a trigger from azure services bus. Integration part was badly created by the function. The connection string was missing related to the queue and in the "Integrate" tab the connection was empty (but not topic and subscription...). After, adding again the connection from "integrate" it worked.

sprabhus commented 5 years ago

@MysticWolf : thanks for sharing your experience, but In my case function app has both http and service bus trigger and service bus connection string are available with the proper integration working condition.

rtaylor72 commented 5 years ago

I am also having this error and I have "AzureWebJobsStorage" specified.

Error:
Internal Server Error - https://integrationhubcorefunction.azurewebsites.net/admin/host/systemkeys/_master 
Session Id: 53df7b45cc2f4a83a50449316e46a9ec
Timestamp: 2018-12-10T17:15:29.650Z
alrod commented 5 years ago

@rtaylor72 I saw storage related errors in logs around 2018-12-1017:30:00 GMT

*System.InvalidOperationException : Secret initialization from Blob storage failed due to a missing Azure Storage connection string. If you intend to use files for secrets, add an App Setting key * 'AzureWebJobsSecretStorageType' with value 'Files'

Microsoft.WindowsAzure.Storage.StorageException : This request is not authorized to perform this operation.

I looks like you fixed this issue. Latest error was in function code itself: Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: CreateTransaction ---> System.ArgumentNullException : Value cannot be null.

alrod commented 5 years ago

@sprabhus I saw an error in the logs:

System.ArgumentException : Value for the connection string parameter name [Guid] was not found.
Parameter name: connectionString

I looks like you fixed this issue.Latest error is Timeout value of 00:30:00 was exceeded by function: Re..nc Can you confirm that?

sprabhus commented 5 years ago

Yes, you are right. I figure out that connection string for one of the function was missing and updated that last week.

Inspite of that I am still receiving the same error,

**Error:

Internal Server Error - https://oneprofilesyncappdev.azurewebsites.net/admin/host/systemkeys/_master Session Id: 79ae11f203a444898151cd3a88ecf319

Timestamp: 2018-12-12T09:42:50.502Z**

alrod commented 5 years ago

@sprabhus, can you please open "Blob Containers" node in Azure Storage Explorer in your storage account. Do you see any containers?

sprabhus commented 5 years ago

@alrod Yes, I am able to connect to the storage account and i don't see any blob container in that account. Please find the screen shot for the same.

storagecontainer

TechnologyAnimal commented 5 years ago

Same problem for me... I, too, have "AzureWebJobsStorage" defined.

Error:

Internal Server Error - https://prod-des-healthcheck.azurewebsites.net/admin/host/keys Session Id: 70dc211828e2463197daf4b5f70a74a4

Timestamp: 2018-12-30T05:43:34.760Z

Edit: It turns out that I was trying to use the azure function 2.0 runtime for powershell, which is not supported (even as experimental). Doing so consistently gives me that error message shortly after the function app deployment. After migrating over to the 1.0 runtime and I have not seen that error again.

thomas-schreiter commented 5 years ago

Same problem here. I use Azure Functions v2 with slot swapping (i.e. deploy to staging slot then swap the slots.)

Internal Server Error - https://expsummarysandbox-func-dev-wus.azurewebsites.net/admin/host/keys Session Id: 486410cb3fa54f1d969e6a86234833f6

Timestamp: 2019-01-04T19:23:01.216Z

More specifically, I just upgraded the Functions version from v1 to v2 when this issue started occurring. What's funny is that the error occurs only on the production slot on alternating deploys -- i.e. on deploy n production is broken and staging works, on deploy n+1 both production and staging work, on deploy n+2 production is broken again and so forth.

Is there a way to hard-reset the keys, e.g. by deleting a blob or something like that? If nothing else works, I will have to recreate the whole function app, but I rather avoid that because I have a couple of environments where I will have to repeat this process.

I tried deploying to the production slot directly, but the error still occurs.

Any ideas, @paulbatum ?

paulbatum commented 5 years ago

@thomas-schreiter this information is really useful, thank you. Just to confirm, in your case does each deployment consist of deploying to staging and then doing a swap?

Yes you can hard delete keys and force them to be regenerated. By default they are in a blob container called 'azure-webjobs-secrets' in the storage account used by your function app. If you are sharing that storage account across multiple function apps, be careful as there will be multiple folders within that container. The host.json file within the folder has host level keys, and then there are json files per function.

thomas-schreiter commented 5 years ago

Thanks for the quick response, @paulbatum. I deleted the host.json. It got recreated after startup, but the errors still occurs.

About your question: Our build pipeline first deploys to staging and then swaps the slot. Now, for debugging this issue, I tried deploying to production directly (i.e. without slot swapping), but it didn't resolve the issue.

thomas-schreiter commented 5 years ago

@paulbatum : I looked further, but didn't find too much. Maybe this bit helps you to find the root cause: I store secrets in a Key Vault and set app settings with a string like "@Microsoft.KeyVault(SecretUri=https://secret_uri_with_version". In Kudu Environment, these strings don't get resolved in the broken production slot. In the staging slot, they do get resolved. Furthermore, the production slot tries to start up but quickly fails when trying to connect to the Azure Service Bus, most likely because the secret wasn't retrieved from the Key Vault.

paulbatum commented 5 years ago

Interesting - this sort of matches up with one of the two errors I see in the logs on my side:

System.ArgumentNullException : Value cannot be null.
Parameter name: uriString
   at System.Uri..ctor(String uriString)
   at Microsoft.Azure.ServiceBus.ServiceBusConnection.InitializeConnection(ServiceBusConnectionStringBuilder builder)
   at Microsoft.Azure.ServiceBus.ServiceBusConnection..ctor(String namespaceConnectionString,TimeSpan operationTimeout,RetryPolicy retryPolicy)
   at Microsoft.Azure.ServiceBus.ServiceBusConnection..ctor(String namespaceConnectionString)
   at Microsoft.Azure.ServiceBus.Core.MessageReceiver..ctor(String connectionString,String entityPath,ReceiveMode receiveMode,RetryPolicy retryPolicy,Int32 prefetchCount)
   at Microsoft.Azure.WebJobs.ServiceBus.MessagingProvider.GetOrAddMessageReceiver(String entityPath,String connectionString) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.ServiceBus\MessagingProvider.cs : 99
   at Microsoft.Azure.WebJobs.ServiceBus.MessagingProvider.CreateMessageProcessor(String entityPath,String connectionString) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.ServiceBus\MessagingProvider.cs : 47
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener..ctor(String entityPath,ServiceBusTriggerExecutor triggerExecutor,ServiceBusOptions config,ServiceBusAccount serviceBusAccount,MessagingProvider messagingProvider) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.ServiceBus\Listeners\ServiceBusListener.cs : 35

I didn't know what to make of that but based on what you mentioned it seems likely that there is some sort of slots + keyvault bug here. Going to see if I can spend 20 minutes trying to repro this now.

Not sure what to make of the other error I'm seeing. This looks like a disposal bug/race in the functions host:

DryIoc.ContainerException : Container is disposed and should not be used: Container is disposed.
You may include Dispose stack-trace into the message via:
container.With(rules => rules.WithCaptureContainerDisposeStackTrace())
   at DryIoc.Throw.It(Int32 error,Object arg0,Object arg1,Object arg2,Object arg3) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 8976
   at DryIoc.Container.ThrowIfContainerDisposed() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 411
   at DryIoc.Container.WithCurrentScope(IScope scope) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 437
   at DryIoc.ResolverContext.OpenScope(IResolverContext r,Object name,Boolean trackInParent) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 2689
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope(IServiceScopeFactory rootScopeFactory) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\ScopedResolver.cs : 40
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs : 101
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesFeature.get_RequestServices()
   at Microsoft.AspNetCore.Http.DefaultHttpContext.get_RequestServices()
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass4_0.<CreateActivator>b__0(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass5_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next,Scope& scope,Object& state,Boolean& isCompleted)

One possibility is that this is a knock-on effect from the first bug. Perhaps our exception handling code has a race or cleanup error. I think it makes more sense to try to get to the bottom of the first error and then see if the second continues.

paulbatum commented 5 years ago

I have set up a repro environment, but I am getting some other type of error when I do a swap (not the same as what you're getting). I'm trying to figure out if I've screwed something up myself or if its a different bug. Probably won't get to the bottom of this until next week.

thomas-schreiter commented 5 years ago

I tried another thing without success: Deploy the old Azure Functions v1 again and then deploy v2. Same error. I noticed after deploying v1, that the secrets still don't get resolved. @paulbatum, let me know if you want to look further into this today. Otherwise I'm going to delete the app and start from scratch to unblock myself.

paulbatum commented 5 years ago

Go ahead and unblock yourself. I have the logs from the errors that your app was encountering and I'm still trying to get a repro.

thomas-schreiter commented 5 years ago

I deleted the app and the underlying storage account and then recreated them both. At first, the error occurred again. Then I realized that the Key Vault needs to authorize to the new app, because it got a new app id (same name, but different id). Everything works now. Looks like everything is working now.

I have a feature request: If the secrets in the app settings can't be resolved, then return a error message saying that, instead of the error message of OP. In my case the function app wasn't authorized to read secrets from the Key Vault.

thomas-schreiter commented 5 years ago

I looked a little bit more. I believe it originally broke when I tried to swap slots that were on different versions. Production was on v1, Staging was on v2, and swapping them somehow messed things up. What does seem to work is first updating both slots individually to v2 -- then it's possible to swap again.

paulbatum commented 5 years ago

Ahh, thats interesting. I assume when you set this up, you made sure that FUNCTIONS_EXTENSION_VERSION wasnt a sticky setting, and therefore its value swapped as part of the swap operation? Because as long as that is done correctly, I feel like this scenario should work (and perhaps there is a bug that we need to find and fix to make it so that it does work).

thomas-schreiter commented 5 years ago

That's right. The app settings including FUNCTIONS_EXTENSION_VERSION are set in the CD pipeline via a Powershell script. All except for one setting are non-sticky -- and the sticky one is not related to the Functions framework.

paulbatum commented 5 years ago

So I figured out what was wrong with my attempt to repro. I was hitting a different issue due to a config mistake I made. I am able to successfully do swaps between two slots for a function app that is using app setting references to key vault and is using MSI to authenticate with the key vault, without any errors.

This backs up the theory we were discussing above, that its related to mixed versions and is not a MSI + Key Vault + Slots issue.

deadwards90 commented 5 years ago

EDIT: Turns out this was a dodgy API route I had put in! Sorry for muddying the waters.

I've got this problem, but I do not have any slots or keys.

App name is ukwest-bill-proc-7e56185c9cd1.

Session id is 4232786c13c6463d982c35f1032b5423

It's a durable function any also have a Service Bus Trigger like the one above.

I've tried completely recreating all resources from scratch (storage, plan and function app) thinking something had got messed up along the way but no dice there.

Happy to provide any more information.

ivana61 commented 5 years ago

I am getting this error: Error: Not Found - https://xxxx.azurewebsites.net/admin/host/keys

I am creating the resource through an ARM template with a release pipeline. The host keys cannot be found. Any idea how can I solve this? I can provide more info if needed.