search

Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.92k stars 442 forks source link

Metadata generation failed. Exit code: '-1073741502' Error: '' #6769

Open RaleS opened 3 years ago

RaleS commented 3 years ago

Getting error when building a project that contains reference to

Is there an easy way to get more information reg. the error?

2020-10-08T08:08:06.8413744Z        Done executing task "Move".
2020-10-08T08:08:06.8413995Z        Task "GenerateFunctions"
2020-10-08T08:08:06.8414279Z          Function generator path: 'C:\Program Files\dotnet\dotnet.exe'
2020-10-08T08:08:06.8415061Z          Microsoft.NET.Sdk.Functions.Generator.dll "D:\agent3work\1548\s\Engines\Engines.Functions\obj\Debug\netcoreapp3.1\PubTmp\Out\bin\Engines.Functions.dll " "D:\agent3work\1548\s\Engines\Engines.Functions\obj\Debug\netcoreapp3.1\PubTmp\Out\ " "False "
2020-10-08T08:08:07.2199809Z        Done executing task "GenerateFunctions".
2020-10-08T08:08:07.2200556Z   26:6>Done building target "_GenerateFunctionsAndCopyContentFiles" in project "Engines.Functions.csproj".
2020-10-08T08:08:07.2209104Z   26:6>Target "_GenerateFunctionsExtensionsMetadataPostPublish" in file "C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.3\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets" from project "D:\agent3work\1548\s\Engines\Engines.Functions\Engines.Functions.csproj" (target "_FunctionsPostPublish" depends on it):
2020-10-08T08:08:07.2210926Z        Using "GenerateFunctionsExtensionsMetadata" task from assembly "C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.3\build\..\tools\netstandard2.0\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.dll".
2020-10-08T08:08:07.2211552Z        Task "GenerateFunctionsExtensionsMetadata"
2020-10-08T08:08:07.2223825Z          Extensions generator working directory: 'C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.3\tools\netstandard2.0\..\netstandard2.0\generator'
2020-10-08T08:08:07.2225792Z          Extensions generator path: 'C:\Program Files\dotnet\dotnet.exe'
2020-10-08T08:08:07.2227810Z          Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.Console.dll "D:\agent3work\1548\s\Engines\Engines.Functions\obj\Debug\netcoreapp3.1\PubTmp\Out\bin" "D:\agent3work\1548\s\Engines\Engines.Functions\obj\Debug\netcoreapp3.1\PubTmp\Out\bin\extensions.json"
2020-10-08T08:08:07.2267963Z          Debug output from extension.json generator:
2020-10-08T08:08:07.2271291Z          
2020-10-08T08:08:07.2558738Z ##[error]C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.3\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(52,5): Error : Metadata generation failed. Exit code: '-1073741502' Error: ''
2020-10-08T08:08:07.2571820Z   26:6>C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.3\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(52,5): error : Metadata generation failed. Exit code: '-1073741502' Error: '' [D:\agent3work\1548\s\Engines\Engines.Functions\Engines.Functions.csproj]
2020-10-08T08:08:07.2572978Z        Done executing task "GenerateFunctionsExtensionsMetadata" -- FAILED.
2020-10-08T08:08:07.2573559Z   26:6>Done building target "_GenerateFunctionsExtensionsMetadataPostPublish" in project "Engines.Functions.csproj" -- FAILED.
2020-10-08T08:08:07.2574321Z   26:6>Done Building Project "D:\agent3work\1548\s\Engines\Engines.Functions\Engines.Functions.csproj" (default targets) -- FAILED.
2020-10-08T08:08:07.2574876Z  37:13>Done executing task "MSBuild".
RaleS commented 3 years ago

Somehow this information got stripped. Project is referencing to following package: <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.3" />

brettsam commented 3 years ago

Can you move Microsoft.NET.Sdk.Functions up to version 3.0.9? There have been a number of fixes since 3.0.3 and this may already be fixed.

If this still repros -- can you then explicitly add a package reference to Microsoft.Azure.Webjobs.Script.ExtensionsMetadataGenerator version 1.2.0? That has even more substantial changes that will be coming soon in Microsoft.NET.Sdk.Functions version 3.0.10 -- but you can try it today with a direct package reference.

RaleS commented 3 years ago

Cool, I have bumped and added

    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.2.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.9" />

I will let you know the results. What is the ETA on 3.0.10?

RaleS commented 3 years ago

Here is the result

2020-10-08T15:12:08.1977635Z   25:6>Done executing task "GenerateFunctions".
2020-10-08T15:12:08.1978068Z   25:6>Done building target "_GenerateFunctionsPostBuild" in project "Engines.Functions.csproj".
2020-10-08T15:12:08.1982850Z   25:6>Target "_GenerateFunctionsExtensionsMetadataPostBuild" in file "C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.2.0\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets" from project "e:\AgentE1\_work\868\s\Engines\Engines.Functions\Engines.Functions.csproj" (entry point):
2020-10-08T15:12:08.1984180Z        Using "GenerateFunctionsExtensionsMetadata" task from assembly "C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.2.0\build\..\tools\netstandard2.0\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.dll".
2020-10-08T15:12:08.1984802Z        Task "GenerateFunctionsExtensionsMetadata"
2020-10-08T15:12:08.2006108Z          Extensions generator working directory: 'C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.2.0\tools\netstandard2.0\..\netstandard2.0\generator'
2020-10-08T15:12:08.2006877Z          Extensions generator path: 'C:\Program Files\dotnet\dotnet.exe'
2020-10-08T15:12:08.2007632Z          Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.Console.dll "e:\AgentE1\_work\868\s\Engines\Engines.Functions\bin\Debug\netcoreapp3.1\bin" "e:\AgentE1\_work\868\s\Engines\Engines.Functions\bin\Debug\netcoreapp3.1\bin\extensions.json"
2020-10-08T15:12:08.2052582Z          Debug output from extension.json generator:
2020-10-08T15:12:08.2053143Z          
2020-10-08T15:12:08.2321157Z ##[error]C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.2.0\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(37,5): Error : Metadata generation failed. Exit code: '-1073741502' Error: ''
2020-10-08T15:12:08.2335240Z   25:6>C:\Nuget\Packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.2.0\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(37,5): error : Metadata generation failed. Exit code: '-1073741502' Error: '' [e:\AgentE1\_work\868\s\Engines\Engines.Functions\Engines.Functions.csproj]
2020-10-08T15:12:08.2336151Z        Done executing task "GenerateFunctionsExtensionsMetadata" -- FAILED.
2020-10-08T15:12:08.2336579Z   25:6>Done building target "_GenerateFunctionsExtensionsMetadataPostBuild" in project "Engines.Functions.csproj" -- FAILED.
2020-10-08T15:12:08.2337167Z   25:6>Done Building Project "e:\AgentE1\_work\868\s\Engines\Engines.Functions\Engines.Functions.csproj" (default targets) -- FAILED.
brettsam commented 3 years ago

The exit code translates to 0xC0000142 which is STATUS_DLL_INIT_FAILED.

Some things on StackOverflow seem to indicate that running as a service may be the issue:

Could that be the issue?

RaleS commented 3 years ago

The error is caused by a hanging dotnet.exe process from previous run. Workaround is to nuke dotnet.exe process and retry build.

JosXa commented 3 years ago

Running into the same error using Microsoft.NET.Sdk.Functions 3.0.9, doesn't seem to be fixed here. I am running this on Jenkins in Windows Server 2019.

Nuking dotnet.exe from the pipeline script doesn't work, however restarting the whole build agent machine does the trick... which obviously I don't want to rely on. Is there an old version I can fall back to where this issue hasn't been present?

deployment\AppName\bin\Develop\netcoreapp3.1\ " "False "
       _GenerateFunctionsExtensionsMetadataPostBuild:
         Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.Console.dll "somelongpath\bin\Develop\netcoreapp3.1\bin" "somelongpath\bin\Develop\netcoreapp3.1\bin\extensions.json"
         Debug output from extension.json generator:

    38>somelongpath\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.8\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(37,5): error : Metadata generation failed. Exit code: '-1073741502' Error: ''

The empty string in the error message is not very helpful :/

Here is the full list of package references, if that helps

``` xml ```

Bakanych commented 3 years ago

We have the same issue when building solution with dotnet publish on private build host with parallel builds. Extremely annoying...

the-castman commented 3 years ago

Any news on this? I am getting the same error but with extensionsmetadatagenerator 1.1.8 and with Microsoft.NET.Sdk.Functions 3.0.9

[error]C:\Windows\ServiceProfiles\NetworkService\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.1.8\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(37,5): Error : Metadata generation failed. Exit code: '-1073741502' Error: ''.

I am getting this on my Azure Devops build pipeline and very sporadic och sudden. Using the clean option on the pipeline doesn´t help, This started to appear like a week ago. Sometimes after waiting a while, triggering a new build it would succeed.

martinnormark commented 3 years ago

We're seeing the same problem, started happening late November 2020.

Version 3.0.11 of Microsoft.NET.Sdk.Functions.

hhhuut commented 3 years ago

Same problem here, started happening this week since we changed something in our Azure Pipelines.

One agent job is to build 3 functions using dotnet publish, one project usually goes through, the other two fail. Microsoft.NET.Sdk.Functions package is on version 3.0.11 for all three functions.

martinnormark commented 3 years ago

@brettsam

Can you move Microsoft.NET.Sdk.Functions up to version 3.0.9? There have been a number of fixes since 3.0.3 and this may already be fixed.

If this still repros -- can you then explicitly add a package reference to Microsoft.Azure.Webjobs.Script.ExtensionsMetadataGenerator version 1.2.0? That has even more substantial changes that will be coming soon in Microsoft.NET.Sdk.Functions version 3.0.10 -- but you can try it today with a direct package reference.

We're using Microsoft.NET.Sdk.Functions v3.0.11, and as per the release notes of v3.0.10 it does contain the changes you're referring to from v1.2.0 of package Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.

We're using these package references in our functions app:

<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.3.1" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="4.2.0" />
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="4.0.3" />
<PackageReference Include="Microsoft.Extensions.Configuration" Version="3.1.8" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.11" />

The output from the publish command ends with:

C:\Windows\ServiceProfiles\NetworkService\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.2.0\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(37,5): error : Metadata generation failed. Exit code: '-1073741502' Error: '' [D:\w3\3\s\Schafwerfen\Schafwerfen.Functions\Schafwerfen.Functions.csproj]

I do believe this is due to a hanging dotnet process as mentioned in https://github.com/Azure/azure-functions-host/issues/6769#issuecomment-707264696 and it seems to hang randomly.

I do not see any evidence of the build or test steps not completing in Azure Pipelines, however it still leaves a dotnet process hanging for some reason.

lewinskimaciej commented 3 years ago

We started having the same issue after switching to dotnet publish from Visual Studio build task in Azure DevOps. It seems that doesn't happen when using publish with .csproj patterns, only when we build entire solution at once(though it would require more testing). Building solution is over 2 times quicker than .csproj pattern with p:BuildInParallel though so we'd like to stick to dotnet publish *.sln. It also happens quite randomly, not on every build. We're also using SDK 3.0.11.

@Edit Also worth mentioning, this started happening when we switched to VM with self-hosted Agent and parallel builds. Now I'm checking it and it seems that if there's only one agent and one build at a time, everything seems ok.

daveaglick commented 3 years ago

FWIW we are also experiencing this exact same problem (and have been for several months). We're on Microsoft.NET.Sdk.Functions 3.0.11, building on Azure Pipelines with local agents hosted on Azure VMs via dotnet build (not dotnet publish), and intermittently get the following:

C:\Users\ba-admin\.nuget\packages\microsoft.azure.webjobs.script.extensionsmetadatagenerator\1.2.0\build\Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.targets(37,5): error : Metadata generation failed. Exit code: '-1073741502' Error: ''

Like others have reported, it seems highly correlated with multiple concurrent builds. We almost never (maybe actually never) see the problem when one build is executing. When two builds are executing on the same build server it seems to happen frequently.

staterium commented 3 years ago

Linking my issue here in case the two are related, or in case my resolution helps anybody else:

7071

chris1411 commented 3 years ago

Same issues here. Creating separate accounts for each build agent isn't a very feasible workaround, especially when you are running 10+ agents. Is this going to be looked at anytime soon?

andyTheCoder commented 3 years ago

Same issue here... Microsoft.NET.Sdk.Functions 3.0.11

dehghanfar commented 3 years ago

this fixed my issue:

To uninstall dotnet core from macOS:

download dotnet-uninstall-pkgs.sh from https://github.com/dotnet/sdk/blob/main/scripts/obtain/uninstall/dotnet-uninstall-pkgs.sh make dotnet-uninstall-pkgs.sh executable execute dotnet-uninstall-pkgs.sh as root (requires superuser privileges to run). curl -O https://raw.githubusercontent.com/dotnet/sdk/main/scripts/obtain/uninstall/dotnet-uninstall-pkgs.sh chmod u+x dotnet-uninstall-pkgs.sh sudo ./dotnet-uninstall-pkgs.sh

basically uninstalling dotnet

c-lamont commented 3 years ago

This has crippled me and made me think about my lifes choices. Any fix for this?

martinnormark commented 3 years ago

@c-lamont This is what worked for us.

Our setup

We run our own agents, because a few years back the performance of the hosted agents were dramatically reduced. Our PR build times more than doubled.

So we have 1 large Windows based VM running in Azure, where we installed 5 build agents. They were installed on C drive, with working folder on D drive (temp storage in Azure VMs).

Fix

I think it was suggested here to assign each service with their own service account.

By default they all run as NETWORK SERVICE, so I created 5 user accounts with a password that never expires, and doesn't require change on next login, made them a member of a security group that has the same name as the agents in the Services snap-in.

It is important to find the correct group, as they grant access to the folder with the binaries on drive C, which gives capability of self updating etc. Folders for me was C:\a1, C:\a2 and so on. (you can also manually grant the appropriate access to the individual accounts).

The new service accounts needs to be assigned to the agents in Services.

Then stop all services, delete the working folders (in my case D:\w1, D:\w2 and so on).

The start all services again, and they should work as expected without this problem.

dotnet tools

This change did something to the discovery of globally installed dotnet tools.

We use dotnet ef commands and un-install and then install the tool as part of the build pipeline:

dotnet tool uninstall -g dotnet-ef
dotnet tool install -g dotnet-ef --version 3.0.0

The dotnet ef commands stopped working, as it couldn't find the tool even though it was globally installed.

We fixed it by adding --tools-path to the (un)-install commands:

dotnet tool uninstall dotnet-ef --tool-path "C:\dotnet tools"
dotnet tool install dotnet-ef --version 3.0.0 --tool-path "C:\dotnet tools"

And then added C:\dotnet tools to the system wide PATH.

c-lamont commented 3 years ago

@martinnormark Thanks for the detailed steps of how you resolved the issue. However my issue was running it locally on VS Mac. I'm sure your response will help out others though.

For me, with the latest VS Mac update Version 8.10.9 (build 3) I do not get the error anymore.

andyTheCoder commented 2 years ago

Seems to be working now... Updated to Microsoft.NET.Sdk.Functions 3.0.13 However, our build server (Jenkins on windows server) was also patched with framework updates (via standard windows updates) So cannot be sure as to which fixed the issue for us.

ksadralodabai commented 1 year ago

Seems to be working now... Updated to Microsoft.NET.Sdk.Functions 3.0.13 However, our build server (Jenkins on windows server) was also patched with framework updates (via standard windows updates) So cannot be sure as to which fixed the issue for us.

I am confirming that we started getting this issue on our Jenkins build servers. Two branches that contained the same code were providing inconsistent results. One was building fine whereas the other one had the error stated in this chain. I tried restarting Jenkins but had the same issue. Finally, after upgrading Microsoft.NET.Sdk.Functions from 3.0.11 to 3.0.13 it got resolved. Hope this helps some folks out there.