az functionapp deployment source config-zip keeps retrying even though deployment failed with 503 Service Unavailable

jurjenoskam commented 4 years ago

On azure-cli 2.0.80 (did not try other versions), publishing code to a Linux Function App will cause the command to keep retrying even though publishing the code failed with 503 Service Unavailable:

$ az functionapp deployment source config-zip --resource-group example-61-linux --name functionappasdwq61 --timeout 180 --src function.zip
Setting ENABLE_ORYX_BUILD to false
Setting SCM_DO_BUILD_DURING_DEPLOYMENT to false
Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 503
Deployment status endpoint https://functionappasdwq61.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...
Deployment status endpoint https://functionappasdwq61.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...
  [...]

My expectation was that the command would fail as soon as it notices that the deployment endpoint responds with status code 503, as there's no sense in retrying at that point.

(To be clear, this issue is not about the deployment failing with 503 Service Unavailable; it is about how Azure CLI handles the 503.)

ghost commented 4 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @ahmedelnably, @fabiocav

fabiocav commented 4 years ago

One of the proposals is to expose an argument to enable you to specify the maximum number of retries, as depending on the deployment, a bad response may occur with the initial requests.

jurjenoskam commented 4 years ago

You mean specifiying the number of retries of the zipDeploy API call? That would indeed be helpful, since the Function App API unfortunately has a tendency to perform some background work after returning a result, causing subsequent calls to fail until "enough" time has passed...

(I read your comment at first as "enabling to specify the max number of retries of getting the deployment status" which would not really help in this situation.)

infrageek commented 4 years ago

Looking fwd to find a solution to this.

ttjackott commented 4 years ago

Same here, this is happening to us quite a bit intermittently, forcing us to use other methods of deploying functions

kbooth-insight commented 4 years ago

Facing the same problem, sometimes it works, other times it doesn't.

ilanRosenbaum commented 4 years ago

same issue here

schaijkw commented 4 years ago

I am also running into this issue. After the zip deploy I got a 503 and the deployment is stuck on the malformed data message.

BG4WCE commented 4 years ago

I am running into the same issue also.

ivan-sam commented 4 years ago

The same problem. When I create a new empty function app using any method (tried just portal web UI) on ASE and try to navigate to its scm endpoint it returns 503. It can take about 10 minutes before this endpoint is available. ASE runs on test environment with Isolated pricing tier:

I1
210 total ACU
3.5 GB memory
Dv2-Series compute equivalent

But this issue never happened before.

UPDATE: It turned out to be a performance issue - App Service Plan had almost 100% CPU and Memory usage.

xostap commented 4 years ago

Experiencing the same issue. In our case, the endpoint returns 404.

Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 404
Deployment status endpoint https://<func-app-name>.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...
...
...

csmcallister commented 4 years ago

Chiming in to say that I, too, am experiencing this issue:

$ az functionapp deployment source config-zip -g resourcegroupname -n funcname --src .\build\functions.zip
$ Getting scm site credentials for zip deployment
$ Starting zip deployment. This operation can take a while to complete ...
$ Deployment endpoint responded with status code 503
$ Deployment status endpoint https://funcname.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...
$ Deployment status endpoint https://funcname.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...
$ Deployment status endpoint https://funcname.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...
.....

sonic1981 commented 4 years ago

I can also confirm that this seems to be a resource issue. Upgrading my service plan fixed it.

Particularly it seems to be an issue with maxed out CPU

you can see above where I upgraded (drop in CPU). The second spike is my performing a release. So releasing seems to be quite CPPU intensive and if you don't have enough CPU it hangs. Could really do with better messaging around this TBH

Splaxi commented 3 years ago

I hit the same issue.

I added the --timeout 360 to my command and it seems to have made a difference.

devu1997 commented 3 years ago

I am observing this for successful deployment with status 202

exinmusic commented 3 years ago

We started to experience this in a CI pipeline where it made for a little deployment roulette. What helped us was bumping our dev environment service plan from a Standard Linux S1 runtime to an S2 (medium). The tier is twice the price, however its solving the issue currently and speeding up our zip deploys today (which is actually a super added value). I'm curious if we might have been just maxing out a limit for the tier perhaps?

The behavior that I could notice from our logs were as follows: If the initial zip deploy endpoint returns a 202, even if the following polling returns malformed for a minute, it will eventually return successful. If the initial deployment endpoint returns any sort of 500 though, the following async calls seem to endlessly return malformed data and never succeeds.

This is the CLI command we run in the pipeline for context. az webapp deployment source config-zip --resource-group --name --src

anatter commented 3 years ago

Having the same issue with status code 400.

Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 400
Deployment status endpoint https://<app-name>.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...

esmcelroy-rci commented 3 years ago

Same thing happens for me with a 403 error. Unlikely it's an access issue, as the service principal I'm using has contributor access on the resource group.

az functionapp deployment source config-zip -g <...> -n <...> --src <...>.zip
WARNING: Getting scm site credentials for zip deployment
WARNING: Starting zip deployment. This operation can take a while to complete ...
WARNING: Deployment endpoint responded with status code 403
WARNING: Deployment status endpoint https://<...>.scm.azurewebsites.net/api/deployments/latest returns malformed data. Retrying...

output of az --version:

az --version
WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'

Please let us know how we are doing: https://aka.ms/azureclihats
and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
azure-cli                         2.25.0 *

core                              2.25.0 *
telemetry                          1.0.6

esmcelroy-rci commented 3 years ago

Solved my issue - it was a Firewall issue. It would be nice if the error was more descriptive!

nagyben commented 2 years ago

Just experienced this issue with az webapp up - fixed it by increasing the tier of the App Service Plan from F1 to B1

Might help in case anyone else experiences this issue in the future. YMMV

Azure / azure-cli

az functionapp deployment source config-zip keeps retrying even though deployment failed with 503 Service Unavailable #11980