Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.95k stars 442 forks source link

Identity got switched off after slot swap failed #4098

Open thomas-schreiter opened 5 years ago

thomas-schreiter commented 5 years ago

By "Idendity" I mean this feature: image

My app got run through our release pipeline, including a slot swap. The slot swap was not successful and eventually timed out after about 10 minutes. This problem is that the Identity of the production slot got switched off. This means, the app could not connect to Key Vault anymore, so no secrets could be retrieved and the app was therefore essentially unusable. What I would expect is that the Identity is unchanged after a failed slot swap and stays on so that the production slot keeps running.

Investigative information

Repro steps

Provide the steps required to reproduce the problem:

I could repro the problem with my specific app:

  1. Verify that Identity is switched on.
  2. Start swapping staging with production slot
  3. This swap did not succeed and timed out in my case. *
  4. Check Identity.

*The reason of the swap timeout is unknown to me. I could repro it when running my CI/CD pipeline, as well as doing the swap manually via the portal. Usually a slot swap takes less than 2 minutes. I have swapped the slots of this app and similar apps many times before without any problems.

Expected behavior

Identity is still on after a failed slot swap.

Actual behavior

Identity got switched off after the failed slot swap

Known workarounds

Switch on Identity manually after the failed swap attempt. The id is the same as before.

Related information

Excerpt from the build log

2019-02-15T17:03:08.0392309Z Swapping App Service 'myfunctionapp' slots - 'staging' and 'production' 2019-02-15T17:03:08.0394379Z ##[debug][POST]https://management.azure.com/subscriptions/5ee94e32-e95c-4110-b707-c95a5f78737d/resourceGroups/myresourcegroup/providers/Microsoft.Web/sites/myfunctionapp/slots/staging/slotsswap?api-version=2016-08-01 2019-02-15T17:03:10.7233301Z ##[debug][GET]https://management.azure.com/subscriptions/5ee94e32-e95c-4110-b707-c95a5f78737d/resourceGroups/myresourcegroup/providers/Microsoft.Web/sites/myfunctionapp/slots/staging/operationresults/5c6ffe3c-b071-422e-bd08-7b458abf1be9?api-version=2016-08-01 2019-02-15T17:03:10.8724639Z ##[debug]Response status code : 202 2019-02-15T17:03:25.8892024Z ##[debug][GET]https://management.azure.com/subscriptions/5ee94e32-e95c-4110-b707-c95a5f78737d/resourceGroups/myresourcegroup/providers/Microsoft.Web/sites/myfunctionapp/slots/staging/operationresults/5c6ffe3c-b071-422e-bd08-7b458abf1be9?api-version=2016-08-01 2019-02-15T17:03:26.0408675Z ##[debug]Response status code : 202 ... ... (a lot more of these) ... 2019-02-15T17:11:34.6328307Z ##[debug][GET]https://management.azure.com/subscriptions/5ee94e32-e95c-4110-b707-c95a5f78737d/resourceGroups/myresourcegroup/providers/Microsoft.Web/sites/myfunctionapp/slots/staging/operationresults/5c6ffe3c-b071-422e-bd08-7b458abf1be9?api-version=2016-08-01 2019-02-15T17:11:34.7766126Z ##[debug]Response status code : 202 2019-02-15T17:11:49.7840090Z ##[debug][GET]https://management.azure.com/subscriptions/5ee94e32-e95c-4110-b707-c95a5f78737d/resourceGroups/myresourcegroup/providers/Microsoft.Web/sites/myfunctionapp/slots/staging/operationresults/5c6ffe3c-b071-422e-bd08-7b458abf1be9?api-version=2016-08-01 2019-02-15T17:11:50.0057658Z ##[debug]Response status code : 400 2019-02-15T17:11:50.0064437Z ##[debug]Completed action

CaioCavalcanti commented 4 years ago

Just had the same problem here, swap operation failed at some point (I found the error below on the activity logs) and the identity of the staging slot was lost. Did you manage to fix this problem?

   [...]
    "operationId": "b984304d-66a7-42ee-b8a4-be68c87fbf0c",
    "operationName": {
        "value": "Microsoft.Web/sites/slots/slotsswap/action",
        "localizedValue": "Swap Web App Slots"
    },
   [...]
    "properties": {
        "statusCode": "InternalServerError",
        "serviceRequestId": null,
        "statusMessage": "{\"Code\":\"InternalServerError\",\"Message\":null,\"Target\":null,\"Details\":[{\"Message\":null},{\"Code\":\"InternalServerError\"},{\"ErrorEntity\":{\"Code\":\"InternalServerError\",\"Message\":null}}],\"Innererror\":null}"
    },