Function App error on startup: WorkerConfig for runtime: custom not found

[ottosulin]

Description of issue

We deploy our Function Apps with Bicep templates. Our Function Apps are custom handlers (Go binary).

Whenever we deploy the Bicep template, this error occurs. I have tested at this point probably hundreds of different configuration combinations and it doesn't seem to make any difference to the result. This occurs consistently with all our Function Apps, which have very similar configurations.

This issue occurs in both of these scenarios:

• Custom handler deployed directly on Function App
• Custom container (occurs with all function app base images) deployed from ACR running the same custom handler

This error has been referenced before, mainly with dotnet-isolated it seems (e.g. this and this).

Also with one of our function apps, currently deployed as a custom container, we're seeing this error which I have not been able to reproduce elsewhere:

Microsoft.Azure.WebJobs.Script.ExternalStartupException : Error building configuration in an external startup class. ---> System.IO.IOException : The file '/home/data/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/3.24.0/extensions.csproj' already exists. at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo,String path,Boolean isDirectory,Func`2 errorRewriter) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path,OpenFlags flags,Int32 mode) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath,FileMode mode,FileAccess access,FileShare share,FileOptions options,Int64 preallocationSize) at System.IO.Strategies.OSFileStreamStrategy..ctor(String path,FileMode mode,FileAccess access,FileShare share,FileOptions options,Int64 preallocationSize) at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream,String path,FileMode mode,FileAccess access,FileShare share,Int32 bufferSize,FileOptions options,Int64 preallocationSize) at System.IO.Compression.ZipFileExtensions.ExtractToFile(ZipArchiveEntry source,String destinationFileName,Boolean overwrite) at System.IO.Compression.ZipFileExtensions.ExtractRelativeToDirectory(ZipArchiveEntry source,String destinationDirectoryName,Boolean overwrite) at System.IO.Compression.ZipFileExtensions.ExtractToDirectory(ZipArchive source,String destinationDirectoryName,Boolean overwriteFiles) at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName,String destinationDirectoryName,Encoding entryNameEncoding,Boolean overwriteFiles) at async Microsoft.Azure.WebJobs.Script.ExtensionBundle.ExtensionBundleManager.DownloadExtensionBundleAsync(String version,HttpClient httpClient) at /src/azure-functions-host/src/WebJobs.Script/ExtensionBundle/ExtensionBundleManager.cs : 176 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.ExtensionBundle.ExtensionBundleManager.GetBundle(HttpClient httpClient) at /src/azure-functions-host/src/WebJobs.Script/ExtensionBundle/ExtensionBundleManager.cs : 110 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.ExtensionBundle.ExtensionBundleManager.GetExtensionBundlePath(??) at /src/azure-functions-host/src/WebJobs.Script/ExtensionBundle/ExtensionBundleManager.cs : 79 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.ExtensionBundle.ExtensionBundleManager.GetExtensionBundleBinPathAsync() at /src/azure-functions-host/src/WebJobs.Script/ExtensionBundle/ExtensionBundleManager.cs : 267 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.DependencyInjection.ScriptStartupTypeLocator.GetExtensionsStartupTypesAsync() at /src/azure-functions-host/src/WebJobs.Script/DependencyInjection/ScriptStartupTypeLocator.cs : 117 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Azure.WebJobs.Script.DependencyInjection.ScriptStartupTypeLocator.<.ctor>b__10_0() at /src/azure-functions-host/src/WebJobs.Script/DependencyInjection/ScriptStartupTypeLocator.cs : 53 at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode) at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication,Boolean useDefaultConstructor) at System.Lazy`1.CreateValue() at Microsoft.Azure.WebJobs.Script.DependencyInjection.ScriptStartupTypeLocator.GetStartupTypes() at /src/azure-functions-host/src/WebJobs.Script/DependencyInjection/ScriptStartupTypeLocator.cs : 68 at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseExternalConfigurationStartup(IWebJobsConfigurationBuilder builder,IWebJobsStartupTypeLocator startupTypeLocator,WebJobsBuilderContext context,ILoggerFactory loggerFactory) at D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 362 at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c__DisplayClass7_3.<AddScriptHostCore>b__8(IWebJobsStartupTypeLocator locator) at /src/azure-functions-host/src/WebJobs.Script/ScriptHostBuilderExtensions.cs : 253 End of inner exception

How to reproduce

1. Create a Bicep template to deploy a custom handler Function App. Configuration options don't really matter, as long as Azure accepts it as valid.
2. Deploy the template, then the function itself with e.g. func azure functionapp publish
3. Then deploy the Bicep template again.
4. Function App fails to start and throws this error.
5. Now you can get it running again by redeploying the function itself with func azure functionapp publish. Sometimes requires 2-3 deploys before the runtime recovers.

[ravitejaavv]

hi @ottosulin have you got any solution for above mentioned issue?

[pranabpm]

@ottosulin any solution you got for this error? My Cx is facing same issue with custom handler too. any pointers to resolve this would help.

[ottosulin]

We migrated to deploying containers on Functions using Azure Container Registry, and that kind of resolved the issue. When deploying we still get some errors when the container comes up but it does run eventually after a minute or so. The container deployment to Functions is not well documented which made it hard to get it working. We tried so many configuration combinations so to be honest I don't remember anymore what helped.

We tried to get help from Azure Support but they had no clue what's happening. I suggest you also open a support request as well and hope you will have better luck there than I had, because Microsoft doesn't seem to be interested in these Github issues.

[dave-biz]

If it helps, I have a demo project as well as rust script to deploy that demo project here.

And proof that it executes within an Azure Function here.

FWIW I've gotten 3x Rust projects deployed to Azure Functions, and each time I've gone through an extended period of somewhat unhelpful error messages such as that which prompted this thread.

And each time the error messages gave me no indication that the project would ever execute correctly until bam! My 50th wild guess gets it running. So I would encourage anyone encountering this thread to keep at it, if only for a little while longer.

And to the Azure Functions team - anything that can be done to improve error messages around Custom Handlers that fail to initialize would be a huge benefit.