Continuous deployment from VSTS no longer works

[michaeldaw]

I deploy to Azure Functions using the "Azure App Service Deploy" task in VSTS. Until recently (I noticed it a couple days ago), it worked fine. Recently, however, the behavior has changed. When a new deployment is uploaded, the actual executing code does not seem to change. For example, if a new HTTP function is added, attempting to call it results in a "404 Not Found" error. This is in spite of the fact that the new function in question is visible in the portal under the "Functions" section of the Function App blade.

If I stop and start the function app, the new code takes effect.

I've seen this behavior in two separate function apps, sometimes when adding new functions to the app, and sometimes when changing code in existing functions.

[jeffhollan]

Looking into this with the VSTS team

[davidebbo]

Can you share your Function App Name, so we can better investigate?

[vincent1173]

@michaeldaw , can you please share the logs for the release? Also, are you using Linux agent to deploy to Function App?

[michaeldaw]

Sorry for the dealy. @jeffhollan For the function app in question: Execution Id: da10f88a-d624-41f7-be5b-8a2abd8b6dc3 UTC Execution Time: 2018-06-27T14:56:29 Region: West US 2

@vincentdass I'm attaching the logs for three releases to that function app on the day I noticed this happening. I hope they are what you're looking for. I didn't see anything out of the ordinary.

ReleaseLogs_78.zip ReleaseLogs_81.zip ReleaseLogs_82.zip

[vincent1173]

@davidebbo @jeffhollan , I couldn't find any error in the logs. Currently, we use MSDeploy to deploy the package. Is there any other info required from VSTS?

[jeffhollan]

Thanks @vincentdass and not sure - @davidebbo let me know if you (or someone else) can look into this or best next step for investigation.

[davidebbo]

I think the next step is to identify whether there is really a deployment issue (i.e. wrong files get deployed), or whether it's more of a runtime issue (deployment works but something goes wrong at runtime). @michaeldaw, see https://github.com/projectkudu/kudu/wiki/Deployment-vs-runtime-issues for some tips on isolating between the two.

[michaeldaw]

@davidebbo Thanks for the link. I don't believe it's a deployment problem. This problem has happened again for me on a different Azure Function App. I used the Kudu console to check the files in the wwwroot folder, and everything looks correct.

As before, the function exists in the Azure Portal, but generates a 404 if accessed. Another, previously existing function in the app continues to work fine.

I've attached the deployment log. You can see that a function is being added and that the files appear to be copied normally.

Below is the information to identify the function app that is currently showing the problem. 0f2eadf1-ab1e-437a-9ebf-1d29e92b9451 2018-06-27T20:41:55.582 West US

I can leave it in this state for a little while. I'll try not to restart the Function App before you have a chance to take a look at it. The new function I attempted to add is called "GetPlotAnalogs". An existing function called "getPlotMeters" continues to work. The functions are both protected using the "Master" function key, but the new one will still result in a 404 if accessed from postman or similar. Accessing the existing function will result in a 401.

P.S.: I also tried running the function manually from the portal. I received this error:

tasklog_3.log

[davidebbo]

I see a NullReferenceException is your logs, coming from one of your functions. Could that be related?

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: getPlotMeters ---> System.NullReferenceException : Object reference not set to an instance of an object.
   at async Scadavantage.Geo.Requests.getPlotMeters.Run(HttpRequestMessage req,TraceWriter log)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) 
   End of inner exception

[davidebbo]

Another error I see that may be more directly related:

Failed to load type 'Scadavantage.Geo.Requests.GetPlotAnalogs' from 'D:\home\site\wwwroot\bin\Scadavantage.Geo.dll'

That's likely why the function is 404, since it doesn't load properly. You should be seeing those same errors in App Insights.

[michaeldaw]

It certainly seems like the last error you mentioned is related. But if the type isn't present in the assembly, why does the function show up in the portal? The .dll file and .json file were built and deployed together. Does this suggest a deployment problem?

[michaeldaw]

I just now tried running the function manually from the portal. The "Failed to Load Type" error no longer occurs. I accessed the function from postman, and it is now working properly. I myself didn't make any changes during this time. Is it possible there's some kind of delay in the deployment process?

[davidebbo]

This looks more like some kind of runtime glitch with assembly loading. Presumably, just restarting the app would have resolved it. @fabiocav may have more to say about it as he's dealt with various assembly loading issues.

[fabiocav]

The function would still show in the portal as that is driven by the metadata in the function.json files. It looks like, somehow, the assembly was found, but there was an issue loading the type from it, which indicates a version mismatch.

Is the Scadavantage.Geo.dll the same assembly you have your functions defined in?

This is with pre-compiled functions using the VS tooling, correct?

It looks like there was an outdated version of the assembly loaded.

[michaeldaw]

Restarting the Function App does solve the problem, but having to do that is not ideal when developing a continuous integration system.

Scadavantage.Geo.dll is the same assembly that the functions are defined in, yes. They are pre-compiled with the VS tooling, yes.

[jeffhollan]

Just cleaning up some old issues and now that v2 is GA confirming this isn't happening anymore?

[michaeldaw]

I haven't seen this problem for quite some time. If something similar occurs in the future, I can pursue it independently of this issue.

Thanks for following up.

[GobiSankarv]

I am getting this error. Failed to load type 'AuditLogFunction.ExceptionLog' from 'D:\home\site\wwwroot\bin\AuditLogFunction.dll'. I just created a function app in VS2017 and deployed thru release pipe line. Function app has only one function and it works fine in the local.

Restarting app did not resolve the issue.

[ThomasArdal]

@davidebbo I think this is still a problem. I'm having the exact same issue as described in the initial bug report. After an Azure DevOps deployment, I see the new dll in the site/wwwroot/bin directoy (i decompiled it to see if the new code was there). But when running my function, it is the old code running. In my case, I don't see any exceptions, since the old code ran without problems as well. After a restart of the app, the new dll is picked up. I can recreate this in a function app I have deployed. I also tried waiting 8 hours to see if that fixed the problem, but the old code was still executed. Restart and it worked.

[jeffhollan]

Can you confirm you are using the recommended “run from package” feature

https://docs.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package

[ThomasArdal]

@jeffhollan I'm not using the "run from package" feature. I'm using the "Deploy Azure App Service" task in Azure DevOps, which uses msdeploy and writes the following in the log:

Updating App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"0"}

[jeffhollan]

The latest version of that actions (4.* I believe?) should have an option to RunFromZip or run from package. That should deploy the way that helps prevent this. There’s also some additional improvements to devops actions rolling out imminently as well to make this even easier

[rasmuschristensen]

@jeffhollan Is this documented somewhere else besides in this thread?

[ThomasArdal]

@jeffhollan I just switched my release definition and it works! Maybe this should be the default in Azure DevOps going forward? Under all circumstances, thank you very much for your help.

For anyone else visiting this issue from Google, here is what I changed:

I also had to change the way I build and drop the package in the build definition as explained here: https://github.com/Microsoft/azure-pipelines-tasks/issues/5259#issuecomment-407804662

[jeffhollan]

🎉 @dariagrigoriu as an FYI but I believe this is the plan:

Maybe this should be the default in Azure DevOps going forward?

[rasmuschristensen]

A small note to the steps you make in the build pipeline. If you forget to uncheck the Prepend root folder name to archive paths The Run from package deployment will work fine, and the zip package also gets deployed to the SitePackages location. But your functions will never be able to run as there now is this extra folder level before the functions files are located. Maybe a note at the deployment task should be added to highlight this when running with Run From Package. I spent many hours solving this issue and only found it thanks to @ThomasArdal 🍺

[jeffhollan]

Thanks - that's something @dariagrigoriu can likely help track as we look to how we can simplify / document / template build and release actions for functions.

[michaeldaw]

Using Deploy Azure App Service V4 with the Run from Package option enabled also solved a strange problem I was having where after a deployment completed, HTTP functions would always result in an Internal Server Error (500) until the Function App was restarted, with no explanation or cause that I could see. After selecting this option, this problem stopped occurring.

[ThomasArdal]

@rasmuschristensen you should comment with a link to your blog post here. I think it would help people coming to this thread from Google.

[rasmuschristensen]

Good idea! You can read it here Azure Function - Run From Package deployment