Dotnet-isolated Function does not start with the latest Extensions.Storage package

plamber commented 2 months ago

Description

Since the last Upgrade to Microsoft.Azure.WebJobs.Extensions.Storage 6.6.0 the Azure Functions Linux in an app hosted plan are throwing this exception during startup.

Timestamp : 9/4/2024 4:44:44 AM
Inner Exception Type: Unable to load startup extension 'AzureStorageQueues' (Type: 'Mi
Total Occurrences: 9
Latest Exception Message: Unable to load startup extension 'AzureStorageQueues' (Type: 'Microsoft.Azure.WebJobs.Extensions.Storage.AzureStorageQueuesWebJobsStartup, Microsoft.Azure.WebJobs.Extensions.Storage.Queues, Version=5.3.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'). The type does not exist. Please validate the type and assembly names.

Unable to load startup extension 'Startup' (Type: 'Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader.Startup, Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader, Version=1.0.0.0, Culture=neutral, PublicKeyToken=551316b6919f366c'). The type does not exist. Please validate the type and assembly names.

No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. Azure Storage, ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. builder.AddAzureStorage(), builder.AddServiceBus(), builder.AddTimers(), etc.).

Steps to reproduce

deploy a dotnet-isolated azure function (linux) with .NET 8.0 with an app service plan (west europe)
create a queue using the Microsoft.Azure.Functions.Worker.Extensions.Storage 6.6.0 package
deploy the code
the function will not load

Update

You find the root cause and solution here: https://github.com/Azure/azure-functions-dotnet-worker/issues/2683#issuecomment-2330536296

plamber commented 2 months ago

Closing the issue. After a decommissioning and redeploy of the Azure Function infrastructure everything started to work normally

plamber commented 2 months ago

Need to reopen this. All our Azure functions stopped working after the deployment of these new bits

plamber commented 2 months ago

We were able to fix the problem by wiping out the function DLLs and downgrading to 6.4.0. We haven't tested higher versions

bjorkstromm commented 2 months ago

deploy a dotnet-isolated azure function (linux) with .NET 8.0 with an app service plan (west europe)

create a queue using the Microsoft.Azure.WebJobs.Extensions.Storage 6.6.0 package

Isn't that an extension package for the in-process model? The equivalent package for isolated worker is Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues

plamber commented 2 months ago

Hi @bjorkstromm,

Thank you for your input. You're right; I've updated the reproduction steps accordingly. However, the error message remains unchanged.

We previously deployed the package without issues, but I noticed that there was an update within the last five days: storage-extension-6.6.0.

After redeploying via GitHub Actions, all our Azure Functions stopped working, resulting in the error message mentioned above. We were able to resolve the issue only by downgrading to version 6.4.0 and redeploying through Visual Studio. It appears that the GitHub Actions process was not updating the code correctly.

Thank you again for your help.

bjorkstromm commented 2 months ago

Thanks for the clarification @plamber

It sounds like a similar issue I just started investigating. We are also facing problems were Linux functions suddenly doesn't find any functions anymore. This is why I found this issue.

plamber commented 2 months ago

We encountered this issue on both the Azure Functions Consumption Plan (Windows) and Azure Functions running on an App Service Plan (Linux).

Unfortunately, the problem went unnoticed during our standard upgrade process because Microsoft did not increment the package version number.

What further complicated troubleshooting was that downgrading and redeploying the function didn't resolve the issue. The problem persisted until we manually deleted the existing DLLs in the target function.

Our temporary mitigation action is to downgrade and redeploy the functions by ensuring that these do not contain any DLLs before

bjorkstromm commented 2 months ago

We haven't yet pinpointed what's causing issues for us. But we noticed today that our staging slot didn't find any functions. Trying to roll back didn't help either, which was weird. I'll do some more investigation later today, and I'll definitely look at the packages you mentioned.

jpdillingham commented 2 months ago

We've been dealing with the same bug since yesterday. I doubt it is related to the issue reported in the OP, but the symptoms are the same; no functions found following a zip deployment.

We've been able to find a temporary workaround by deleting the WEBSITE_RUN_FROM_PACKAGE envar following the deployment. We have no idea why this works, but it does for us.

jpdillingham commented 2 months ago

Ignore this if you're not deploying with GitHub actions!

We've tracked the issue down to a change in the behavior of the upload-artifact GitHub action. Discussion here: https://github.com/actions/upload-artifact/issues/602

tl;dr there's a hidden .azurefunctions directory that is no longer being uploaded with v3 or v4 of this action. Removing the WEBSITE_RUN_FROM_PACKAGE envar was a red herring, causing the function to revert to some older files (or something?)

We're back up and running after pinning upload-artifact to v.4.4.0 and setting include-hidden-files: true (requires download-artifact@v4 or higher)

bjorkstromm commented 2 months ago

Ignore this if you're not deploying with GitHub actions!

We've tracked the issue down to a change in the behavior of the upload-artifact GitHub action. Discussion here: actions/upload-artifact#602

tl;dr there's a hidden .azurefunctions directory that is no longer being uploaded with v3 or v4 of this action. Removing the WEBSITE_RUN_FROM_PACKAGE envar was a red herring, causing the function to revert to some older files (or something?)

We're back up and running after pinning upload-artifact to v.4.4.0 and setting include-hidden-files: true (requires download-artifact@v4 or higher)

Thank you @jpdillingham! This was the root cause for us at least :+1: I can't understand how the actions team thought doing this kind of breaking change without a major version bump would be good idea.

satvu commented 2 months ago

@plamber Just want to confirm - is the root cause of your issue also the GitHub Actions change?

Also - appreciate the investigations/contributions here and especially the root cause analysis. Thank you all!

plamber commented 1 month ago

Hello, thank you. I can confirm that this is due to a breaking change of the upload-artifact-v4 GitHub action. Thank you @jpdillingham for bringing this up.

I am going to close this issue. I list down the root cause analysis in case somebody else might come into the same issue.

The issue was not caused by the library updates but rather by a breaking change introduced in upload-artifact v4. This version no longer includes hidden files during the upload process, which are essential for proper Azure Functions deployment when using the specified libraries.

To resolve this, update the actions/upload-artifact@v4 configuration to include the include-hidden-files: true parameter, and ensure that actions/download-artifact@v4 is also being used.

Example:

- name: Upload engine
  uses: actions/upload-artifact@v4
  with:
    ...
    include-hidden-files: true
    ...

We will be patching all workflows in our environment to accommodate this change.

Azure / azure-functions-dotnet-worker