Receiving "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying our app

[mcdurdin]

This issue has been moved over from https://developercommunity.visualstudio.com/content/problem/753520/receiving-error-error-failed-to-deploy-web-package.html

We are sporadically receiving "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying our app, with no other useful details that I can find. Following this, I can usually log in to dev.azure.com and start a new deployment which runs successfully. I just can't find any more detail about what the root issue is or how we can address this problem.

For deployment issues, please provide us with the following information:

Repro steps.

your project built successfully on your dev machine but failed on Azure? please write down your build tools and their versions (ie Msbuild 15.1.0.0)

The project builds successfully both locally and on Azure (Node.js backend / Angular+Typescript front end). It fails only in the Release pipeline.

Project structures.

in order to reproduce your issue at our end we need a simple github repository that highlights structure of the project

See below; the site is straightforward in structure and documented, hopefully fairly minimal as it stands. The site won't run without specific keys for build.palaso.org (and perhaps github.com, although any user's key should work here); I can supply test keys if required.

The log/error given by the failure.

Normally this include a stack trace, error code and some more information.

2019-09-29T20:01:05.5384672Z ##[section]Starting: Deploy Azure App Service
2019-09-29T20:01:05.5523287Z ==============================================================================
2019-09-29T20:01:05.5523400Z Task         : Azure App Service deploy
2019-09-29T20:01:05.5523499Z Description  : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
2019-09-29T20:01:05.5523568Z Version      : 3.4.31
2019-09-29T20:01:05.5523636Z Author       : Microsoft Corporation
2019-09-29T20:01:05.5523703Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-rm-web-app-deployment
2019-09-29T20:01:05.5523797Z ==============================================================================
2019-09-29T20:01:06.3287589Z Got connection details for Azure App Service:'status-keyman-com'
2019-09-29T20:01:07.4387353Z Package deployment using ZIP Deploy initiated.
2019-09-29T20:03:25.3932504Z ##[error]Failed to deploy web package to App Service.
2019-09-29T20:03:25.3976382Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)
2019-09-29T20:03:29.6500803Z ##[warning]Error: Failed to update deployment history. Error: Conflict (CODE: 409)
2019-09-29T20:03:29.6646084Z ##[section]Finishing: Deploy Azure App Service

Debug your Azure website remotely.

it is recommanded that you share your Web App name, directly or indirectly we can take a look at what's going on.

The GH repo is https://github.com/keymanapp/status.keyman.com; live site is https://status.keyman.com

Mention any other details that might be useful.

---

Thanks! We'll be in touch soon.

[suwatch]

@sanchitmehta please investigate this linux deployment issue.

@mcdurdin the 409 conflict indicated ongoing deployment. Please check if you have other deployments active during such time.

[mcdurdin]

Please check if you have other deployments active during such time.

We didn't have any other ongoing deployments as far as I can tell from the logs.

[asmendez]

Hi, I have the same problem, since I have 2 applications in the same app service plan and I have both integrated with Azure devops through the Development Center. Even I have only one pipeline that deploys first to staging and after to production. I don't know what's wrong because a few days ago this was working well.

[sanchitmehta]

@asmendez Did the first deployment fail for some reason ? We have a cool off period in Deployment Locks (when an unexpected error occurs leading to a Kudu Crash) to avoid conflicts. Could you share your app name ?

@mcdurdin are you still facing this problem ?

[mcdurdin]

@mcdurdin are you still facing this problem ?

We have not seen any errors in the last few deployments so I think it's good now.

[asmendez]

@asmendez Did the first deployment fail for some reason ? We have a cool off period in Deployment Locks (when an unexpected error occurs leading to a Kudu Crash) to avoid conflicts. Could you share your app name ?

@mcdurdin are you still facing this problem ?

Hi, at the end I changed to a Windows App Service because in Linux this problem came up even with a fresh and new instance with the CD/CI. Even when I deleted all my instances from the same App Services Plan. I don't know why this was happening but I decided to change to a Windows PHP Web App and I don't have the problem now.

[evelo2]

Sorry, this issue is very much still alive. Existing builds that were deploying successfully are now failing consistently.

I have posted the YAML to StackOverflow

Any help would be appreciated.

[nmaarse]

I have the same problem : Package deployment using ZIP Deploy initiated.

[error]Failed to deploy web package to App Service.

[error]To debug further please check Kudu stack trace URL : https://$dqipowebportal0001:***@portal0001.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace

[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

[guiihlopes]

I had the same problemas some days ago, and the issue is caused sometimes when the azure plan CPU reaches 100%.

It gives conflict error message or this one:

An unknown error has occurred. Check the diagnostic log for details.
##[error]Failed to deploy web package to App Service.

[sanchitmehta]

@nmaarse @evelo2 This error implies that Kudu was already processing a deployment request and to ensure that deployment is an atomic operation Kudu throws 409. We would have to see if the pipelines was sending multiple requests. Could you open a support ticket (thereby you could share the app name) and we can investigate it further.

[nmaarse]

@nmaarse @evelo2 This error implies that Kudu was already processing a deployment request and to ensure that deployment is an atomic operation Kudu throws 409. We would have to see if the pipelines was sending multiple requests. Could you open a support ticket (thereby you could share the app name) and we can investigate it further.

Hi sanchimehta. Happy to hear from you. Could you explain to me how and where to open a support ticket.

[wadeem]

same problem here

[akisvolanis]

If you are on a free subscription plan, make sure you haven't reached your storage limit which is 1gb. If so, you have to upgrade to a paid one..

[i3anaan]

Had the same issue today. The service plan had plenty of room left (CPU/Memory/Disk), no other deployments were being processed at the time and a run that previously succeeded failed when retried today (same commit). Additionally it seemed that kudu crashed as well or something, the suggested link for more debug information gave a 500 and other attempts to access kudu, logs or files of the function all failed (the ones I tried at least).

In the end I managed to resolve the issue by deleting the function app, and re-creating it (same settings). Thankfully this was a (new) function in a development environment, this would be an unacceptable workaround on a live production environment.

[jstolwijk]

Since today all our applications cannot be deployed, some applications receive a 500 while others receive a 409 response. Like @i3anaan pointed out deleting the function app and re-creating it resolves the issue. We cannot do this in prod since it will result in tens of minutes of down time

[jjavierdguezas]

same issue since today here (same scenario than @i3anaan ), any solution other than recreating everything?

[jjavierdguezas]

I have solved the problem by scaling up the service plan where my azure functions were. I have switched from B2 to B3 tier, which only increase RAM and ACUs so I'm guessing that we had consumed all ACUs ?

[IFYates]

This also started happening for us today on S1 tier in multiple App Service Plans. Scaling up to S2 tier solved it. After scaling back down to S1, I was still able to release.

Out typical ASP usage on S1 is 6% CPU and 60% memory.

Interesting note, which may not be relevant: in one ASP, I changed the tier up and then down again without doing a release in between. This did not solve the issue. The workaround has only fixed the issue if I do at least one release to the ASP at the new tier, before reverting it.

Edit: Should say thanks to @jjavierdguezas for the idea of changing tier

[Kenneth-Abrams]

Experienced the same 409 error issue during deployment starting on 7/27 . Function apps on linux ASP.

[inatlewski]

I'm experiencing the same problem. 2020-07-29T15:06:45.5681580Z ##[error]Failed to deploy web package to App Service. 2020-07-29T15:06:45.5701015Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

[benoutram]

Experiencing the same problem here in a DevOps Release pipeline of App Service and Windows Function App projects. We first saw this error occur on 21/07/20 but seeing several occurrences today. The region for both the App Service and Functions is West Europe.

2020-07-29T09:50:50.1288995Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2020-07-29T09:50:50.4010044Z App Service Application settings are already present.
2020-07-29T09:51:46.6646202Z Package deployment using ZIP Deploy initiated.
2020-07-29T09:52:53.1693953Z ##[error]Failed to deploy web package to App Service.
2020-07-29T09:52:53.1707005Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

A possible workaround seems to be to change the deployment method to Zip deploy, but for one of our Application Services this has increased the deployment time from 1 minute to 21 minutes which is not going to be acceptable.

[jjavierdguezas]

Lets see if they can do something... https://twitter.com/jotajota930621/status/1288503705994764290

[pazcaldj]

Had this issue also several times today... Retried it a couple of minutes ago and succeeded... Seems like its fixed?

[mahmoudhammouda]

Hello , we faced this issue several time today and we used the webdeploy as workaround.

Le mer. 29 juil. 2020 à 19:19, Pascal de Jonge notifications@github.com a écrit :

Had this issue also several times today... Retried it a couple of minutes ago and succeeded... Seems like its fixed?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/projectkudu/kudu/issues/3042#issuecomment-665793402, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL67I6UGR4WEPSOCLLIKBATR6BK2JANCNFSM4I3ZILCQ .

[zorge]

Having the same issue on S1, after upgrading the apps plan to a higher tier the issue is gone and the build succeeds. This is of course is not acceptable, Microsoft!

[suwatch]

I would be great if you all can share the webapp name - this would help us pinpoint.

[grantwar]

I'm having the same issue with multiple function apps, and they are hosted in a P2V2 and in another case in an S1.

[sanchitmehta]

@grantwar Are you still experiencing the same problem ? Could you provide us your app name indirectly ?

[grantwar]

@grantwar Are you still experiencing the same problem ? Could you provide us your app name indirectly ?

@sanchitmehta Yes I'm still experiencing the issue, the app is failing when I'm trying to deploy to both my P2V2 and S1 service plans.

I have another app name if you need it

[jjavierdguezas]

@suwatch I sent you an email with the information to the address in your profile, but since I scaled up the app service we have not seen the error anymore

[sanchitmehta]

@jjavierdguezas Was your app deployed in West Europe ?

[jjavierdguezas]

@jjavierdguezas Was your app deployed in West Europe ?

yes they are

[zorge]

Ive also had my service deployed in West Europe and after scale-up the error was gone. But I'd like to scale down and know what the error was caused by. Unfortunately I cannot give the app name here.

[suwatch]

Please follow this to report site name https://github.com/projectkudu/kudu/wiki/Reporting-your-site-name-without-posting-it-publicly

@jjavierdguezas I did receive your message - but can't make anything out of the image. Please share the site name (here or indirectly) as well as the UTC time of the incident. We will investigate

[jjavierdguezas]

Please follow this to report site name https://github.com/projectkudu/kudu/wiki/Reporting-your-site-name-without-posting-it-publicly

@jjavierdguezas I did receive your message - but can't make anything out of the image. Please share the site name (here or indirectly) as well as the UTC time of the incident. We will investigate

In my email I listed the six apps affected,

The subscription id pattern: 58747***-****-****-****-***f432a26a1

We noticed the incident on July 29 around 8:45 am UTC We did the "fix" around 1:00 pm UTC

Let me know if you need more info

[suwatch]

From our telemetry, we observed below exception happening to multiple sites and vms started a few days ago. We are unable to repro. If anyone is still stuck in this state, please try to go to portal, click restart the site (assuming you can afford site to be down for such short period). Try zipdeploy again - and see if that mitigate the issue.

Error occurred type="error" text="The type initializer for '<Module>' threw an exception." stackTrace="   at Newtonsoft.Json.Converters.BinaryConverter.CanConvert(Type objectType)
   at Newtonsoft.Json.JsonSerializer.GetMatchingConverter(IList`1 converters, Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.InitializeContract(JsonContract contract)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.CreateObjectContract(Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.CreateContract(Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.ResolveContract(Type type)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.Linq.JToken.FromObjectInternal(Object o, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.Linq.JObject.FromObject(Object o, JsonSerializer jsonSerializer)
   at Kudu.Core.Infrastructure.LockFile.WriteLockInfo(String operationName, Stream lockStream) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\LockFile.cs:line 213
   at Kudu.Core.Infrastructure.LockFile.Lock(String operationName) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\LockFile.cs:line 153" innerText="Attempted to read or write protected memory. This is often an indication that other memory is corrupt." innerStackTrace="   at <CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
   at .cctor()

[suwatch]

Update

We are currently investigating. It is likely due to recent update on the VMs leading to this. It affects "West Europe", "West US 2", "North Central US" and "West Central US" regions.

You can mitigate yourself by performing either of the below.

1) Restarting Kudu process. Browse to SCM site -> Process Explorer and Kill w3wp (SCM). The new w3wp will start up automatically. If you have multiple instances, you may need to use instance dropdown to iterate and do the same on all instances.

2) Goto portal. Navigate to site blade, click Restart. This unfortunately will cause short down time for the site.

3) Scale up and scale down. This indirectly restarts SCM sites.

We are also gradually mitigate and will provide ETA once we know it. Apology for the inconvenience.

[i3anaan]

@suwatch Good to know there is being worked on an a fix! Would it still help you if we shared more information about the apps being affected?

I think this information should give you the ability to pinpoint one of our function apps affected by this issue:

• 2020-07-30T08:02:12Z [Information] Id=eec0c97e-8138-4bd9-81fe-cd233ec4dedf)

(The guide/steps in the docs seems lightly outdated)

[jjavierdguezas]

Today we faced this issue in another subscription, this time I restarted the app (as @suwatch said) and then the release went just fine. So, no scaling is really needed... good news to save money... 😅

[julen-folky]

I had the issue too. I could not kill the w3wp process because Kudu did not show anything in the process' list. Restarting the site did not work. I stop the site, deploy again (using Azure DevOps pipelines), start again. Worked! Then I tried a new deploy without stopping the site. Also works.

The problem and steps above executed on staging slot. Production slot was not affected by the issue.

[lewis-harris]

I had the issue too, in west europe.. I could not kill the w3wp process because Kudu did not show anything in the process' list. Restarted the site and deployed again succesfully

[julen-folky]

FWIW, @lewis-harris I have the site in west europe too.

[remcorakers]

We received the same issue since Friday last week for one of our App Services in West Europe, using Zip-Deploy. We tried to do the following:

1. Restart the App Service and re-deploy our code via our Azure build pipeline => still received the 409 error
2. Stop the App Service re-deploy our code via our Azure build pipeline => still received the 409 error
3. Kill all w3wp processes in the SCM console and re-deploy our code via our Azure build pipeline => still received the 409 error
4. Delete the App Service, re-deploy the App Service via our ARM templates, stop the App Service, and re-deploy our code via our Azure build pipeline => success!

So only deleting the entire App Service and re-deploying fixed the issue.

[kayhantolga]

Had the same issue ** Scaled up app service, stop and wait a bit, run again, run the deployment => worked fine

I tried scaling up and down then deployment, didn't work for me.

[yoelOhayon]

Same issue , could not kill the w3wp process because Kudu did not show anything in the process list. restart the app changed nothing

I added a new slot and configure my Azure Devops Release to the new slot and it worked !

[sympthom]

This is all of the sudden a problem for us (this has not been a problem since 1.5 years). Why do we get "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying to WE using zip deploy? How can we do to fix it? This is not a descibable error message.

[suwatch]

@sympthom did you try the mitigation mentioned above?

[atmike]

I have the same problem. but i could solve it by stopping the app-service, starting it again and then restarting it 3 times in quick succession. This seems to stop the w3wp and the release is working again.

[Gismork]

We started experiencing this issue on one App Service on July, 29th hosted in West Europe. We are now seeing this issue on other App Service.

We were able to workaround this issue by killing the Kudu process and redeploying our application again on the App Service. We use Azure DevOps to deploy our applications (using the Run from package deployment mode).

[abercea]

I have the same problem. but i could solve it by stopping the app-service, starting it again and then restarting it 3 times in quick succession. This seems to stop the w3wp and the release is working again.

This also worked for me (azure func on win, run from package deploy)