search

Azure / Azure-Functions

1.11k stars 195 forks source link

Managed identity not changed after swap slots operation #2511

Open samuel-kogler-AP opened 2 months ago

samuel-kogler-AP commented 2 months ago

Description

We sometimes encounter an issue that after a swap slots operation, the production slot is running using the managed identity of the deployment slot. When this happens, we restart the function app and everything works again.

The issue happens (but probably is not limited to) with Http and ServiceBus triggers and the CosmosDB and ServiceBus .NET client libraries.

However, that behaviour is problematic for our continuous deployment pipeline, which automatically performs a zip deployment and a subsequent slot swap if the zip deployment was successful.

We are using:

We also tried setting the AzureFunctionsWebHost__hostid setting explicitly for deployment and production slots (but they were unique before anyways). This had no effect.

I suspect the issue is just a missing refresh of the managed identity or a missing cache invalidation.

Let me know if you need more information.

Steps to reproduce

  1. Create function app using latest dotnet-isolated worker
  2. Use managed identity and log/assert it on trigger (e.g. check managed identity and compare to cloud_RoleName/url)
  3. Repeat a couple of times (~ every 3rd to 5th time in our case):
    1. Deploy new zip file az functionapp deployment source config-zip
    2. Swap slots az functionapp deployment slot swap
    3. Trigger function
jenspettersson commented 1 month ago

I just posted about a similar experience but on a regular AppService: https://github.com/Azure/azure-sdk-for-net/issues/45367

It happens every time for us, if we're not adding a Deployment slot setting, then it never happens...

mattchenderson commented 4 weeks ago

Hi - as a general statement for anyone reading this issue, a support ticket would be appropriate for anyone experiencing this in their apps.

Thanks @samuel-kogler-AP for the details on your setup and repro! Could you please confirm if your setup involves deployment slot settings?

@jenspettersson I'm very curious about the comment you made about the presence of a deployment slot setting influencing this. That very well could be a clue to a root cause of the behavior you're seeing. Capturing in this thread an excerpt from that Azure SDK issue:

So to summarize:

  • When swapping slots without any Environment variables marked "Deployment slot setting" everything is swapped, including the Managed Identities.
  • When swapping slots with at least one Environment variables marked "Deployment slot setting" the Managed Identities are not swapped.

I'd be particularly interested to know if others see a distinction in this behavior based on deployment slot settings as well. We'll also see if we can repro that on our end.

jenspettersson commented 4 weeks ago

Hi - as a general statement for anyone reading this issue, a support ticket would be appropriate for anyone experiencing this in their apps.

We have an ongoing support ticket since last friday on this. However, they're having trouble reproducing it on their end so not much progress there yet.

I'd be particularly interested to know if others see a distinction in this behavior based on deployment slot settings as well. We'll also see if we can repro that on our end.

I just tested this again (on our "regular" App Service, haven't tested on an Azure Function App yet but based on what @samuel-kogler-AP is reporting, it might be the same issue) and it was "reproduceable" immediately.

I'm happy to help if I can.