Autoswap issues from GitHub

[neusamir]

Our issue is similar to https://github.com/projectkudu/kudu/issues/1124 but may not be the same so we are opening a new issue.

Setup

• We have a production azure web app
• We have 2 staging slots that are continuously deployed from GitHub
  • dev slot: This pulls in the 'dev' branch from our GitHub repo and deploys. No issues here. Works everytime. Not using autoswap.
  • ppe slot: This pulls in the 'master' branch from our GitHub repo and deploys. This autoswaps to the 'production' slot. This is where we are seeing issues. About 50% of the time the swap fails. Sometimes it fails two times in a row. To fix it, we just update the master branch and that kicks off a new build and that usually fixes it.

    Error

<step title="POST /operations/autoswap?slot=production&amp;operationId=AUTOSWAPce25aabe-d981-412d-bf47-ff440bf9613f, x-ms-request-id: d440ee48-f7d1-4b11-be10-f4dfdf33ecc2" date="2016-06-23T22:09:15.918" >
    <step title="Response: NotFound" date="2016-06-23T22:09:20.005" /><!-- duration: 0ms -->
</step><!-- duration: 4097ms -->
<step title="Error occurred" date="2016-06-23T22:09:20.053" type="error" text="One or more errors occurred." stackTrace="   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
    at Kudu.Services.FetchHandler.&lt;&gt;c__DisplayClass18_1.&lt;PerformBackgroundDeployment&gt;b__2()
    at Kudu.Contracts.Infrastructure.LockExtensions.TryLockOperation(IOperationLock lockObj, Action operation, String operationName, TimeSpan timeout)
    at Kudu.Contracts.Infrastructure.LockExtensions.LockOperation(IOperationLock lockObj, Action operation, String operationName, TimeSpan timeout)
    at Kudu.Services.FetchHandler.&lt;&gt;c__DisplayClass18_0.&lt;PerformBackgroundDeployment&gt;b__1()" innerText="Response status code does not indicate success: 404 (Not Found)." innerStackTrace="   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
    at Kudu.Core.Infrastructure.OperationClient.&lt;PostAsync&gt;d__7`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Kudu.Core.Deployment.AutoSwapHandler.&lt;HandleAutoSwap&gt;d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
    at Kudu.Services.FetchHandler.&lt;PerformDeployment&gt;d__12.MoveNext()" /><!-- duration: 0ms -->
</step><!-- duration: 286964ms -->

Site

• dummy8080.azurewebsites.net
• The site we are having an issue with starts with 'n'

[neusamir]

@shrimpy Thanks for looking at this. When do you think this fix will get pushed to production?

[shrimpy]

Keep this issue open till we deploy.

@neusamir, not yet, but will be soon. Github auto close this issue while i merge that PR

[neusamir]

Ok great, will do.

[shrimpy]

@neusamir i can manually patch your deployment if you want the fix sooner. but you will have to do some manual work once we deploy (when this issue closed)

here is the flow if your want to move forward:

• i will patch your main site and dev slot (since they are the two will be autoswap)
• Later when we deploy, i will update this issue and close it
• You will need to go to both your main site and dev slot with below two url

https://{main site name}.scm.azurewebsites.net
https://{slow site name}.scm.azurewebsites.net

(an easy way is go to portal, click browse, and modify the url, add ".scm")

• Click "Debug Console" at the top of the menu
• Navigate to "D:\home\SiteExtensions\Kudu"
• Rename or Remove "applicationHost.xdt"
• Restart site/slot correspondingly after rename or removal their "applicationHost.xdt" file
• Go to SCM site landing page, you should see a meaning full build version instead of "xx.0.0.0" like below
• (Optional) Go back to debug console, Navigate to "D:\home\SiteExtensions" and remove "Kudu" folder
• After above process, both your main site and dev slot will use the deployment we push out

[neusamir]

Ok, do you have any idea when it will be deployed automatically? 1 week? 1 month? If it will be a long time then we can go through the manual process, otherwise I would like to wait.

[shrimpy]

@neusamir we should have a deployment sometime next week.

[neusamir]

Ok, we will wait.

[shrimpy]

@neusamir there was a deployment last night, your site should be fix now if you restart them.

[neusamir]

Thanks, we had some builds go out last night around 10PM PST and had some weird errors. We will test this will some more builds next week and let you know if we have any issues.