code-simian
commented
11 months ago We've changed the function from Linux to Windows and it instantly worked. This was without any other changes in our pipeline and configuration.
Open Palving opened 2 years ago
code-simian
commented
11 months ago We've changed the function from Linux to Windows and it instantly worked. This was without any other changes in our pipeline and configuration.
richard-berg
commented
11 months ago Unfortunately, it looks like Python functions are not supported on Windows.
richard-berg
commented
11 months ago Completely deleting the resource group, function app, storage, etc was still not enough. Something in the deployment framework is busted.
For help pinpointing, note that I was wrong about it working yesterday; last succesful deploy was actually October 9.
FinVamp1
commented
11 months ago @richard-berg I've reproduced your issue and this is related to setting the TZ app setting which isn't supported on Linux Consumption SKU's this is only supported on dedicated and Elastic Premium SKU's. This is discussed at https://github.com/Azure/azure-functions-host/issues/9203 . I'll be adding Checks to show this as a configuration issue in Diagnose and Solve,
richard-berg
commented
11 months ago Thanks @FinVamp1! I would've never figured that out from the (lack of) deployment logs alone.
I was able to get my app deployed, with the triggers reconfigured to UTC times. They'll be 1hr off during Daylight Savings, but that's better than not running at all.
edonosotti
commented
11 months ago Same problem here. It looks like this one is being open for over a year and it's a real big trouble. I think the issue is at the platform level though, because I found it out while I was re-deploying a Function App because the function suddenly stopped working and no issues are being tracked in the internal metrics and logs. Bad stuff.
tmakin
commented
11 months ago Just hit this deploying a linux function app to consumption plan using the v2 release task in azure devops. I had removed a timer trigger from the function app and the new build failed with the "Failed to sync triggers" error. I was still able to deploy a previous build.
The fix for me was to delete all the AzureWebJobs.YourFunctionName.Disabled config settings relating to old triggers which had been superseded.
lyadeski
commented
11 months ago Has anyone solved this? its been occurring on all our production deployments and our systems are completely offline and we are at a complete loss what to do right now.
8:43:13 PM PROD-XXX: Syncing triggers... 8:43:23 PM PROD-XXX: Syncing triggers (Attempt 2/6)... 8:43:34 PM PROD-XXX: Syncing triggers (Attempt 3/6)... 8:43:54 PM PROD-XXX: Syncing triggers (Attempt 4/6)... 8:44:35 PM PROD-XXX: Syncing triggers (Attempt 5/6)... 8:45:55 PM PROD-XXX: Syncing triggers (Attempt 6/6)... 8:45:56 PM: Error: Encountered an error (InternalServerError) from host runtime.
code-simian
commented
11 months ago @lyadeski have you tried switching the app host to Windows instead of Linux?
lyadeski
commented
11 months ago @lyadeski have you tried switching the app host to Windows instead of Linux?
i did but that introduces new issues. i dont understand what has changed, everything was working fine until a couple days ago, we've made no code changes or infrastructure changes and suddenly everything stops working - very frustrating
hamdrew
commented
11 months ago I encountered this error when I had neglected to configure Azure Private Endpoint DNS. This prevented the Function App runtime from resolving the hostname of the Azure Storage private endpoint.
I had to open a support case with Microsoft to get to my solution. I highly recommend you do the same if you are encountering this error. They have additional views into the engine that can help pinpoint the issue.
tr3ysmith
commented
11 months ago Has anyone solved this? its been occurring on all our production deployments and our systems are completely offline and we are at a complete loss what to do right now.
8:43:13 PM PROD-XXX: Syncing triggers... 8:43:23 PM PROD-XXX: Syncing triggers (Attempt 2/6)... 8:43:34 PM PROD-XXX: Syncing triggers (Attempt 3/6)... 8:43:54 PM PROD-XXX: Syncing triggers (Attempt 4/6)... 8:44:35 PM PROD-XXX: Syncing triggers (Attempt 5/6)... 8:45:55 PM PROD-XXX: Syncing triggers (Attempt 6/6)... 8:45:56 PM: Error: Encountered an error (InternalServerError) from host runtime.
We had the exact same problem yesterday. Function App was completely down, couldn't redeploy, and I open a support ticket with Microsoft to only be told that I either needed to switch away from consumption plan, or implement a timer function to keep the app from going to cold start. Absolutely ridiculous answer to the problem... Sounds to me like theres a bug with the consumption plan currently, and no one wants to address it.
Our function app worked perfectly fine before the weekend, and we made no code changes or configuration changes, it just stopped working.
For now, I moved to the Premium plan, and its back working, but we're going to be looking into alternative options, since we can't have this happen again, and not get proper answers from the support team.
FinVamp1
commented
11 months ago Hello all,
I'm Finbar and I'm part of the Functions engineering team at Microsoft and I work on the features related to Application Code Deployment. As part of the Functions deployment logic we call SyncTriggers endpoint. This makes sure that the Functions Runtime and the underlying infrastructure are up to date with the triggers in the Functions for this Function app.
This thread is long and so I will summarize the symptoms people get and the potential causes for each. If this update doesn't help you resolve the issue then please consider opening a support case or providing me with an app name and timeframe for when your issue occurs.
Summary of issues related to Sync Triggers
More Details This issue was specific to issues with SyncTriggers when ran in the context of the AzureFunctionApp DevOps tasks. I see a number of people are seeing this error from other client tools. The troubleshooting of this error depends on the underlying status code from the Sync Trigger call.
If this doesn't help then please contact Support or provide an app name and timeframe in this thread or email to me finbarr at microsoft.com
We understand that this error can be cryptic and we are looking at how best to resolve this in two areas.
Thanks,
Finbar
telmotrooper
commented
11 months ago Here's what worked for me:
I was trying to zip deploy to a Linux/Node.js-based Function App. I had a working instance created through the Azure Portal and another created via a Bicep template where the ServiceUnavailable error would happen.
I've used the following commands to get the settings for each instance:
az resource show -g "RESOURCE_GROUP_NAME" --resource-type "Microsoft.Web/sites" -n "FUNCTION_APP_NAME_1"
az resource show -g "RESOURCE_GROUP_NAME" --resource-type "Microsoft.Web/sites" -n "FUNCTION_APP_NAME_2"
Diffing the outputs I noticed that my new function app was missing a definition for the linuxFxVersion setting, so I've updated my Bicep template to include that variable and assign the value NODE|18 to it. With that done I was finally able to perform a zip deploy to this instance.
Here's a part of the Bicep template for reference:
resource functionApp 'Microsoft.Web/sites@2022-09-01' = {
name: functionAppName
location: location
kind: 'functionapp'
properties: {
serverFarmId: appServicePlan.id
siteConfig: {
appSettings: [
{
name: 'AzureWebJobsSecretStorageType'
value: 'files'
}
{
name: 'AzureWebJobsStorage'
value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccount.name};AccountKey=${storageAccount.listKeys().keys[0].value};EndpointSuffix=core.windows.net'
}
{
name: 'APPINSIGHTS_INSTRUMENTATIONKEY'
value: applicationInsights.properties.InstrumentationKey
}
{
name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
value: 'InstrumentationKey=${applicationInsights.properties.InstrumentationKey}'
}
{
name: 'FUNCTIONS_WORKER_RUNTIME'
value: 'node'
}
{
name: 'FUNCTIONS_EXTENSION_VERSION'
value: '~4'
}
{
name: 'WEBSITE_RUN_FROM_PACKAGE'
value: '1'
}
]
linuxFxVersion: 'NODE|18'
}
}
}
ozkary
commented
11 months ago I was getting the Syncing Triggers BadRequest error during the deployment. I am using Linux Y1 service plan. The runtime was set to 3.0. I swtiched to 4.0, and there were no more errors. You can access the runtime setting by visiting:
Configuration-> function runtime setting (tab)
Runtime 3.0 - Error
Deployment successful. deployer = Push-Deployer deploymentPath = Functions App ZipDeploy. Extract zip. Remote build.
Remote build succeeded!
Syncing triggers...
Syncing triggers...
Syncing triggers...
Syncing triggers...
Error calling sync triggers (BadRequest). Request ID = '31e425ca-7ce2-4856-b03d-b15771ce0f10'.runtime 4.0 - OK
Remote build succeeded!
Syncing triggers...
Functions in fn*****:
**** - [httpTrigger]
Invoke url: https://***.azurewebsites.net/api/***
jamalsoueidan
commented
10 months ago For me the issue was related to the storage account, the keys in the azure function app setting wasn't correct and pointed to a deleted azure storage accounts, and the application needed that to deploy.
Then it worked!
jamalsoueidan
commented
10 months ago Hello all,
I'm Finbar and I'm part of the Functions engineering team at Microsoft and I work on the features related to Application Code Deployment. As part of the Functions deployment logic we call SyncTriggers endpoint. This makes sure that the Functions Runtime and the underlying infrastructure are up to date with the triggers in the Functions for this Function app. ....
Please see my comment, the health check couldn't see that my settings had storage account that didn't exist.
joskuijpers-imec
commented
10 months ago We have found that we get this error when deployment throws an exception. E.g. when our (tirgger) configuration was wrong. You can find this exception in the Log Stream during deployment. Fixing such exception fixed our deployment.
This is with dotnet-isolated runtime. This seems to be different behavior than we had before.
VladislavsKugelevics
commented
10 months ago We have found that we get this error when deployment throws an exception. E.g. when our (tirgger) configuration was wrong. You can find this exception in the Log Stream during deployment. Fixing such exception fixed our deployment.
This is with dotnet-isolated runtime. This seems to be different behavior than we had before.
Same for me. Also, log stream does not work, so I can't even find which exception do I have. This is infuriating ;-( @joskuijpers-imec do you have any advice how to find out which exception is thrown or how to fix log stream?
joskuijpers-imec
commented
10 months ago @VladislavsKugelevics we only looked at the log stream, sorry to hear that doesn't work for you
VladislavsKugelevics
commented
10 months ago I have advice for anyone having the same issue. If you are not being able to see the error in log stream or application insights.
traces | where message contains "WorkerError" | order by timestamp desc
captainhook
commented
10 months ago We managed to switch the Linux App Service Plan from Y1 (Consumption) to B1 and it resolved it for us. It broke for us after adding AAD auth.
I understand this has been already posted as a solution working for some people and not others. Nevertheless this is the Az CLI we ran:
az login
az account set --subscription "[your sub name]"
az appservice plan update --name "[your app service plan name]" --resource-group "[resource group name for the app service plan]" --sku B1If Microsoft divulges any info on why this is happening, I'll be sure to share with a follow up.
nxp200
commented
9 months ago For others who may be struggling with this error message, I have run into this issue for the past 6 weeks and finally was able to identify what was causing it for my Function App. With all the correct configuration settings set up for vnet integration etc with the storage account, I was still experiencing incosistent deployment success/failures. After opening a ticket with Microsoft, they were able to look at back end logs - not visible to end users during deployment - and there was an error related to the Python packed named "typing". This was in the requirements.txt file we were trying to deploy but we were not aware that package comes default with Python and was not needed to be added. Removal of this package from the requirements.txt file fixed the issue completely and deployments have been 100% successful since.
"Why does this cause Syncing of Triggers to fail sometimes?" I am told by Microsoft support that it installs the package successfully, but after the function app restarts after deployment of code (successfully), it hits an error that prevents the restart from completing in time. Sometimes it progresses past the error fast enough and Syncing of Triggers is able to complete, and other times it doesn't.
"What could be done to help avoid or debug this issue?" Better error logging to the end users during deployment via VS Code deployment logs.
This is my understanding so far, so am happy to be corrected anywhere I may have mis-spoken. This is also likely an isolated incident with the "typing" package, but it might give clues for others out there regarding the requirements.txt file and packages being installed.
hem-sharma
commented
8 months ago I also encountered with this issue recently. But I found below issue in my situation:
It got worked for me after solving those.
Hope this might be helpful to anyone.
MitchellMcKenna
commented
6 months ago I ran into this issue because I had not compiled my custom handler for the correct runtime, for example we are using golang so we had to do:
GOOS=linux GOARCH=amd64 go build -o handler handler.goOnce i did that, sync triggers worked right away with func azure functionapp publish...
JuulJuup
commented
5 months ago We didn't specify the python version in our azure-pipelines.yml file. We fixed this issue by adding the following code in our build step:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'
bergdaniel
commented
1 month ago I'll just chime in here if anybody is still experiencing this problem. It seems that run from package should can't be used with Y1 Linux. Instead you need to use Windows. That's what solved it.
The documentation even explicitly states this.
Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name: AzureFunctionApp@1
list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment
Azure pipelines
Agent - Hosted or Private:
If using Hosted agent, provide agent queue name: Agent name: 'Azure Pipelines 2' Agent machine name: 'fv-az313-661' Current agent version: '2.210.1' Operating System Ubuntu 20.04.5 LTS
If using private agent, provide the OS of the machine running the agent and the agent version:
Issue Description
Deployment to function apps worked in the beginning but has however started to consistently fail across deployments. Even after deletion and re-deployment of the function app, plans and storage account it fails when zip-deploying. The task itself seems to bring the app down, and it takes a while for it to be available again.
Task logs
This is a linux function app I am trying to deploy with consumption plan
Troubleshooting
Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting
Error logs