FBryant87
commented
1 year ago Getting this issue sporadically also, sometimes consistently for long periods of time
Open Palving opened 1 year ago
FBryant87
commented
1 year ago Getting this issue sporadically also, sometimes consistently for long periods of time
somabodonyi
commented
1 year ago I am only getting this for function apps, that use consumption plans.
jonraf
commented
1 year ago Im also getting this issue sometimes.
maQus123
commented
1 year ago Me too (AzureFunctionApp@1, Hosted Agent, Azure Functions Linux Consumption).
Edit: Now, one day later, it worked for whatever reason. Can the action be made more robust please? We need to deploy production soon :D
Palving
commented
1 year ago Suddenly working for us aswell. Would like some clarification around this issue as we cannot know for certain it will not happen again.
maQus123
commented
1 year ago @nadesu Would you be so kind and give an update on how likely it is that the problem will persist or continue to occur over the next week(s)? Thank you very much.
Palving
commented
1 year ago @nadesu Still happening often. Any update? With this error it is unfeasible to use function apps on consumption plans in production environments
joergen98
commented
1 year ago @nadesu Happening for me aswell
jacob7395
commented
1 year ago Hi all,
This is an issue for me too, I managed to resolve it by recreating my storage account, service plan and function app the running the config again.
az functionapp deployment source config-zip -g ***-functions -n ***-identity-functions --src src.zip
Prior to rebuilding I was getting the issue consistently, I had changed my plan from Y1 to B1 a few times and also been doing docker deployment to the same functions app. I wonder if the storage account got some garbage in. When running the deploy with --debug the following error was shown.
DEBUG: msal.application: Broker enabled? False INFO: cli.azure.cli.command_modules.appservice.custom: Syncing Triggers... DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={} DEBUG: cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={} DEBUG: msal.application: Cache hit an AT DEBUG: msal.telemetry: Generate or reuse correlation_id: ae87ec0b-3661-43c6-bd12-80290851f6de DEBUG: cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/***/resourceGroups/***-functions/providers/Microsoft.Web/sites/***-identity-functions/syncfunctiontriggers?api-version=2022-03-01' DEBUG: cli.azure.cli.core.sdk.policies: Request method: 'POST' DEBUG: cli.azure.cli.core.sdk.policies: Request headers: DEBUG: cli.azure.cli.core.sdk.policies: 'Accept': 'application/json' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '40f952a8-5449-11ed-9542-6c9466047e4f' DEBUG: cli.azure.cli.core.sdk.policies: 'CommandName': 'functionapp deployment source config-zip' DEBUG: cli.azure.cli.core.sdk.policies: 'ParameterSetName': '-g -n --src --debug' DEBUG: cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.41.0 (MSI) azsdk-python-azure-mgmt-web/7.0.0 Python/3.10.5 (Windows-10-10.0.19044-SP0)' DEBUG: cli.azure.cli.core.sdk.policies: 'Authorization': '**' DEBUG: cli.azure.cli.core.sdk.policies: Request body: DEBUG: cli.azure.cli.core.sdk.policies: This request has no body DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 DEBUG: urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions//resourceGroups/-functions/providers/Microsoft.Web/sites/-identity-functions/syncfunctiontriggers?api-version=2022-03-01 HTTP/1.1" 400 346 DEBUG: cli.azure.cli.core.sdk.policies: Response status: 400 DEBUG: cli.azure.cli.core.sdk.policies: Response headers: DEBUG: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache' DEBUG: cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache' DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Length': '346' DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8' DEBUG: cli.azure.cli.core.sdk.policies: 'Expires': '-1' DEBUG: cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' DEBUG: cli.azure.cli.core.sdk.policies: 'Server': 'Microsoft-IIS/10.0' DEBUG: cli.azure.cli.core.sdk.policies: 'X-AspNet-Version': '4.0.30319' DEBUG: cli.azure.cli.core.sdk.policies: 'X-Powered-By': 'ASP.NET' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-writes': '1199' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '59a4ae66-0524-47ff-b224-eaf65a87711f' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '59a4ae66-0524-47ff-b224-eaf65a87711f' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'UKSOUTH:20221025T094208Z:59a4ae66-0524-47ff-b224-eaf65a87711f' DEBUG: cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff' DEBUG: cli.azure.cli.core.sdk.policies: 'Date': 'Tue, 25 Oct 2022 09:42:07 GMT' DEBUG: cli.azure.cli.core.sdk.policies: Response content: DEBUG: cli.azure.cli.core.sdk.policies: {"Code":"BadRequest","Message":"Encountered an error (ServiceUnavailable) from host runtime.","Target":null,"Details":[{"Message":"Encountered an error (ServiceUnavailable) from host runtime."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","Message":"Encountered an error (ServiceUnavailable) from host runtime."}}],"Innererror":null} DEBUG: cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception: DEBUG: cli.azure.cli.core.util: Traceback (most recent call last): File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/appservice/custom.py", line 524, in enable_zip_deploy_functionapp File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/appservice/custom.py", line 735, in upload_zip_to_storage File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/appservice/custom.py", line 728, in upload_zip_to_storage File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 73, in wrapper_use_tracer File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/mgmt/web/v2022_03_01/operations/_web_apps_operations.py", line 48499, in sync_function_triggers azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Bad Request' Content: {"Code":"BadRequest","Message":"Encountered an error (ServiceUnavailable) from host runtime.","Target":null,"Details":[{"Message":"Encountered an error (ServiceUnavailable) from host runtime."},{"Code":"BadRequest"},{"ErrorEntity":{"Code":"BadRequest","Message":"Encountered an error (ServiceUnavailable) from host runtime."}}],"Innererror":null}
ERROR: cli.azure.cli.core.azclierror: Operation returned an invalid status 'Bad Request' ERROR: az_command_data_logger: Operation returned an invalid status 'Bad Request' DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x04967B68>] INFO: az_command_data_logger: exit code: 1 INFO: cli.main: Command ran in 17.878 seconds (init: 0.487, invoke: 17.391) INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1 INFO: telemetry.client: Accumulated 0 events. Flush the clients. INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1 INFO: telemetry.save: Save telemetry record of length 3185 in cache INFO: telemetry.check: Returns Positive. INFO: telemetry.main: Begin creating telemetry upload process. INFO: telemetry.process: Creating upload process: "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry__init__.pyc C:\Users\jacob.azure" INFO: telemetry.process: Return from creating process INFO: telemetry.main: Finish creating telemetry upload process.
Palving
commented
1 year ago It's hard to tell where the issue lies, I however attempted calling the synctriggers api directly. https://management.azure.com/subscriptions/*********/resourceGroups/rgName/providers/Microsoft.Web/sites/functionApp/syncfunctiontriggers?api-version=2022-03-01.
It works for the first couple of calls, however after a while it starts to return 400, same as in the deployment, with the same error message as seen in the deployment logs.
I've also noted that the api-version differs from the Azure task, and the latest available, hard to say if it makes any difference.
FabianBigler
commented
1 year ago Any news here of how to fix this? We are having the same issue.
Palving
commented
1 year ago We ended up moving back to app services on windows with dedicated plans.
ihordyrman
commented
1 year ago I am facing the same problem with my new function. Consumption plan, Linux.
vienleidl
commented
1 year ago Changed to B1, it works for me, it seems to only happen on the Linux Y1 Consumption (Serverless) plan.
FinVamp1
commented
1 year ago Hello, I will investigate and come back to this thread.
vienleidl
commented
1 year ago Hello, I will investigate and come back to this thread.
Thank you very much! And I've already tried to remove the app setting, it also works well https://github.com/Azure/functions-action/issues/147#issuecomment-1251143109
ihordyrman
commented
1 year ago Don't know if it will help you, but In my case, the problem was caused because I tried to deploy it while the app was stopped.
sangxxh
commented
1 year ago Changed to B1, it works for me, it seems to only happen on the Linux Y1 Consumption (Serverless) plan.
Have tried both Y1 and B1 Linux both don't work.
What worked for me was to choose Windows OS instead of Linux for the App Service Plan.
pawel-zolty
commented
1 year ago @FinVamp1 isn't it problem with this line in the log:
Value of WEBSITE_RUN_FROM_PACKAGE has been changed to https://dtpmdcfirstrefusaddevsa.blob.core.windows.net/azure-pipelines-deploy/package_1663846829501.zip?***
Run from package is not supported for linux azure functions on consumption app service plan.
Palving
commented
1 year ago It's actually required, according to documentation. https://learn.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package
TechWatching
commented
1 year ago Same error for me. It seems to happen without any reason, without anything that has changed from our side. It's a big blocker, it prevents us from deploying updates to our functions.
FinVamp1
commented
1 year ago Hello, will investigate with priority today. The task deploys using WEBSITE_RUN_FROM_PACKAGE_URL by default for the Linux Consumption plan.
pawel-zolty
commented
1 year ago @FinVamp1 does the task also override deploymentMethod parameter if set to zipDeploy as OP wrote? Can we deploy zipDeploy withot Website_Run_From_Package?
@Palving Are you sure? Here is shown that consumption linux functions support both run from package with URI and simple zip deploy.
Edit: Zip deploy for consumption linux function is supported only when performing remote build:
Linux function apps running in the Consumption plan don't have an SCM/Kudu site, which limits the deployment options. However, function apps on Linux running in the Consumption plan do support remote builds.
When to use it: External package URL is the only supported deployment method for Azure Functions running on Linux in the Consumption plan, if the user doesn't want a remote build to occur. When you update the package file that a function app references, you must manually sync triggers to tell Azure that your application has changed. When you change the contents of the package file and not the URL itself, you must also restart your function app manually.
FinVamp1
commented
1 year ago Hello, currently the task defaults to using the setting WEBSITE_RUN_FROM_PACKAGE=URL. The Linux Consumption plan also supports using WEBSITE_RUN_FROM_PACKAGE=1 and ZipDeploy. This requires enabling your site to use Azure Files by enabling the WEBSITE_AZUREFILES_CONNECTIONSTRING and WEBSITE_CONTENTSHARE. We are thinking about how to add this functionality for the task and still avoid breaking changes.
vienleidl
commented
1 year ago Changed to B1, it works for me, it seems to only happen on the Linux Y1 Consumption (Serverless) plan.
Have tried both
Y1andB1Linux both don't work. What worked for me was to choose Windows OS instead of Linux for the App Service Plan.
It works for both of them, just remove that kind of app setting and try again, then it should work with this app setting: WEBSITE_RUN_FROM_PACKAGE=URL
Mabiro
commented
1 year ago Same error for me. It seems to happen without any reason, without anything that has changed from our side. It's a big blocker, it prevents us from deploying updates to our functions.
@TechWatching
We're currently having the same issue, pipelines aren't not able to deploy without any changes from our end.
Was this resolved for you?
TechWatching
commented
1 year ago Yes Azure had some issues at the moment. It worked when he issues were solved
FinVamp1
commented
1 year ago Hi, I'll close this issue and people can log a new one if they are seeing this issue again.
psulik321
commented
1 year ago Hi there. we are facing this issue again today. 2023-05-03T07:25:41.6828057Z ##[error]Error: Failed to sync triggers for function app 'XXXXXXXXXX'. Error: BadRequest - Encountered an error (ServiceUnavailable) from host runtime. (CODE: 400) It is deployed via zip package directly via Azure DevOps without any templates or anything like that.
fzoccara
commented
1 year ago I had the same issue, on azure function linux based on consumption plan. Not just deployment, if I made a request through postman I get a "The service is unavailable."
WaterKnight1998
commented
1 year ago Same here
jesravn
commented
1 year ago We are having the same issue. An Azure Pipeline deployment of a Function App that used to run, but now throws Error: BadRequest - Encountered an error (ServiceUnavailable) from host runtime. (CODE: 400)
jprochnow
commented
1 year ago Same issue!
SvenRelijveld1995
commented
1 year ago Here as well!
Palving
commented
1 year ago Hi, I'll close this issue and people can log a new one if they are seeing this issue again.
@FinVamp1 Could this issue be reopened?
kyurkchyan
commented
1 year ago Having this same issue arbitrarily with a pipeline that worked perfectly fine a while ago. I am using the following task
- task: AzureFunctionApp@1.221.101
displayName: 'Deploy $(FunctionAppName) Azure Function'
inputs:
azureSubscription: ${{ parameters.AzureServiceConnection }}
appType: functionAppLinux
appName: $(FunctionAppName)
resourceGroupName: $(ResourceGroup)
package: $(Pipeline.Workspace)/$(artifact)/$(artifact).zip
blazyng
commented
1 year ago Same here
Syncing triggers (Attempt 6/6)... Error: Encountered an error (ServiceUnavailable) from host runtime.
blazyng
commented
1 year ago Deployment works when I disable system assigned identity for my application.
tnsholding
commented
1 year ago I can confirm the same issue and resolution by removing the assigned identity.
So some bug must exist when having managed identity assigned and doing deployments?
blazyng
commented
1 year ago Adittionaly user defined identity are not working either. Azure runtime functions is no longer reachable (error in azure portal) when i activate one of the identity types.
Palving
commented
1 year ago Issues with managed identity might explain why the issue persists even after deletion of app and storage account, due to how managed identity caches
Managed identity tokens on the other hand are cached by the underlying Azure infrastructure for performance and resiliency purposes: the back-end services for managed identities maintain a cache per resource URI for around 24 hours. This means that it can take several hours for changes to a managed identity's group or role membership to take effect. Today, it is not possible to force a managed identity's token to be refreshed before its expiry. If you change a managed identity’s group or role membership to add or remove permissions, you may therefore need to wait several hours for the Azure resource using the identity to have the correct access.
FinVamp1
commented
1 year ago Hello, I've not been able to reproduce this in my tests. Can someone share their repro steps or a sample time frame or App Name?
blazyng
commented
1 year ago We are not using Azure any longer. So i can not confirm if the error still persists. Steps to reproduce the problem for me were: Create a fresh Azure Function App, deactivate insights. (Locations tested were Germany and Europe West)
Than > activate user defined identity or system assigned identity for the new function App and i could no longer snyc triggers wiht VSCode and additionaly inside the Azure Portal the Azure function itself created the message "Azure Functions Runtime is unreachable"". So i could also not reach any Trigger or add Triggers from inside the Azure Portal after activating any identiy.
namiskuukkel
commented
1 year ago I hit this error on following situation: I'm doing AZ-204 labs https://github.com/MicrosoftLearning/AZ-204-DevelopingSolutionsforMicrosoftAzure/tree/master on my own machine instead of using the lab VM and I faced this issue when doing labs 2 and 7 which use Azure functions.
On the labs publishing is done with func azure functionapp publish <functionname>.
The labs don't instruct you to set AzureWebJobsStorage app setting on Azure function app configuration prior to doing the publishing. If this value isn't set, the error returned from the publishing command is as follows: Azure Functions Core Tools does not support this deployment path. Please configure the app to deploy from a remote package using the steps here: https://aka.ms/deployfromurl
I ended up setting up remote package deployment and AzureWebJobsStorage on app settings when debugging the problem. With WEBSITE_RUN_FROM_PACKAGE set, I ended up with this trigger syncing error.
In my case the error was solved when I removed WEBSITE_RUN_FROM_PACKAGE app setting. Hence, I got misdirected to this error by the bad error message caused by missing AzureWebJobsStorage app setting from the function app.
UPDATE: It appeared that even tho the trigger syncing "succeeded", the function app didn't have any triggers when checking on Azure Portal. Hence, I re-ran the publishing, and now it's trigger sync is failing again.
svrooij
commented
12 months ago Not sure if it helps, but I was having similar issues and I fixed it (for me) by switching the Azure Functions Deploy to version 2.*.
Before I tried 4 times, all failed, after the switch it completed immediately.
jfellien
commented
11 months ago @svrooij we use version 2 since a while, but ran into issue this days 😢
My guess, the resources at Azure are running out. We deploy mostly to GermanyWestCentral, but not all resources are available there, so we have to move it to a different regions then.
After a few tries. I removed the App Service Plan, the Function App and its related Storage Account. Now it has been deployed.
Our setup:
Consumption-Y1 Function App Zip Deployment via Azure DevOps
- task: AzureFunctionApp@2
displayName: Run Deployment ###
inputs:
azureSubscription: $(###Connection###)
appType: functionAppLinux
appName: '$(###Name)'
package: '$(Pipeline.Workspace)/$(publishArtifactName)/*.zip'
milkyjoe90
commented
11 months ago I've been experiencing this since 6pm last night, linux consumption app returning 503 for both http triggered calls and deployments from Azure DevOps pipelines and releases
code-simian
commented
9 months ago I'm seeing this issue as well since I migrated the function to isolated worker process.
This is a Linux function on a Y1 plan. If I revert the code back to an in process function then the deployment works. As an isolated worker process it fails all the time.
We are deploying via Azure DevOps with
- task: AzureFunctionApp@2
inputs:
azureSubscription: $(azureSubscription)
appType: functionAppLinux
appName: $(appName)
package: "$(Pipeline.Workspace)/**/*.zip"
deploymentMethod: "auto"
runtimeStack: $(runtimeStack)With runtimeStack: "DOTNET|6.0"
MarGraz
commented
9 months ago In our case, with .NET 6, Azure Function v4, Microsoft Identity, the problem was that we had a Function with a reference to the WebApp, used to access the DB Identity context (because we had the necessity to retrieve users when the function was running). Locally it worked perfectly, but after this change the release process was always returning:
Syncing triggers for function app
##[error]Error: Failed to sync triggers for function app 'func-name-dev'. Error: BadRequest - Encountered an error (InternalServerError) from host runtime. (CODE: 400)After spending a lot of time looking for a solution, we decided to remove the WebApp reference from the Function, and we created a library with a separated scaffolding (DB first). The release process started working again.
richard-berg
commented
8 months ago Deploying a Python functions app (v4) from VSCode's built-in integration has started failing with the same error.
I tried removing the managed identity, per the above suggestions, but that did not help. Neither did removing the host.json file from the storage container (as advised in a different issue).
12:10:57 PM cantori-attendance: Resetting all workers for cantori-attendance.azurewebsites.net
12:10:57 PM cantori-attendance: Deployment successful. deployer = ms-azuretools-vscode deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
12:11:10 PM cantori-attendance: Syncing triggers...
12:11:23 PM cantori-attendance: Syncing triggers (Attempt 2/6)...
12:11:34 PM cantori-attendance: Syncing triggers (Attempt 3/6)...
12:11:54 PM cantori-attendance: Syncing triggers (Attempt 4/6)...
12:12:35 PM cantori-attendance: Syncing triggers (Attempt 5/6)...
12:13:55 PM cantori-attendance: Syncing triggers (Attempt 6/6)...
12:13:59 PM: Error: Encountered an error (InternalServerError) from host runtime.I've been trying several times for the last hour. Same app deployed ok yesterday.
Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name: AzureFunctionApp@1
list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment
Azure pipelines
Agent - Hosted or Private:
If using Hosted agent, provide agent queue name: Agent name: 'Azure Pipelines 2' Agent machine name: 'fv-az313-661' Current agent version: '2.210.1' Operating System Ubuntu 20.04.5 LTS
If using private agent, provide the OS of the machine running the agent and the agent version:
Issue Description
Deployment to function apps worked in the beginning but has however started to consistently fail across deployments. Even after deletion and re-deployment of the function app, plans and storage account it fails when zip-deploying. The task itself seems to bring the app down, and it takes a while for it to be available again.
Task logs
This is a linux function app I am trying to deploy with consumption plan
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs