search

microsoft / durabletask-dotnet

Out-of-process .NET SDK for the Durable Task Framework
MIT License
115 stars 33 forks source link

Workaround for "Error loading native library" error on Mac M1/Mac M2 #148

Open davidmrdavid opened 1 year ago

davidmrdavid commented 1 year ago

When trying out the Durable Functions SDK for .NET isolated, Mac M1 / M2 users may encounter the following error:

A host error has occurred during startup operation 'xxx-xxxx-xxxx-xxxx'.
[2023-02-17T16:33:49.933Z] Grpc.Core: Error loading native library.
 Not found in any of the possible locations: [...]/bin/output/.azurefunctions/libgrpc_csharp_ext.arm64.dylib,[...]/bin/output/.azurefunctions/runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib,[...]/bin/output/.azurefunctions/../../runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib.
Value cannot be null. (Parameter 'provider')

This is a temporary limitation and has a known workaround. Please see the instructions below, which were originally contributed by @vdboots and @apavate here.

  1. Add the Nuget package Contrib.Grpc.Core.M1 to the Azure Function Project by either using the Nuget Package Manager or by just adding the following line to the .csproj file.
<PackageReference Include="Contrib.Grpc.Core.M1" Version="2.41.0" />
  1. Add the following line to the .csproj file right before the last tag.
<Target Name="CopyGrpcNativeAssetsToOutDir" AfterTargets="Build"> <ItemGroup> <NativeAssetToCopy Condition="$([MSBuild]::IsOSPlatform('OSX'))" Include="$(OutDir)runtimes/osx-arm64/native/*" /> </ItemGroup> <Copy SourceFiles="@(NativeAssetToCopy)" DestinationFolder="$(OutDir).azurefunctions/runtimes/osx-arm64/native" /> </Target>
  1. Run the project and it should work!

This exception is currently being worked on and we will update this post once resolved.

R00iBaard commented 1 year ago

Excellent stuff!!! Many thanks. Absolute life saver, had a low priority task on pause for the last few months 🥳

colbylwilliams commented 1 year ago

@davidmrdavid @jviau Contrib.Grpc.Core.M1 2.46.6 was recently released. Should people using this workaround update to the newest version? Also Is there an ETA on when this workaround will no longer be needed?

vdboots commented 1 year ago

@colbylwilliams We updated to the newer library without any issues.

LloydNicholson commented 1 year ago

Newest version 2.46.6 doesn't work with workaround anymore for my project.

This is our current Functions.csproj file

<Project>
<ItemGroup>
    ** other package references **

         <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.0.3" />
    <!-- https://github.com/Azure/azure-functions-durable-extension/issues/2446#issuecomment-1517203490 -->
        <PackageReference Include="Contrib.Grpc.Core.M1" Version="2.46.6" />
    </ItemGroup>

    <!-- Workaround for M1 and M2 macs: https://github.com/Azure/azure-functions-durable-extension/issues/2446#issuecomment-1517203490 -->
    <Target Name="CopyGrpcNativeAssetsToOutDir" AfterTargets="Build">
        <ItemGroup>
            <NativeAssetToCopy Condition="$([MSBuild]::IsOSPlatform('OSX'))" Include="$(OutDir)runtimes/osx-arm64/native/*"/>
        </ItemGroup>
        <Copy SourceFiles="@(NativeAssetToCopy)" DestinationFolder="$(OutDir).azurefunctions/runtimes/osx-arm64/native"/>
    </Target>
</Project>

This currently breaks with version 2.46.6 with error 

Grpc.Core: Error loading native library "[...]/bin/Debug/net7.0/.azurefunctions/runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib". .
Value cannot be null. (Parameter 'provider')

but 2.41.0 still works.

@vdboots do you have the exact implementation? I am curious how yours works and mine isn't working.

colbylwilliams commented 1 year ago

@LloydNicholson I got the same error with 2.46.6. 2.41.0 still works.

chuongxl commented 1 year ago

Confirm that version 2.41.0 works well, But 2.46.6 breaks with the error "Grpc.Core: Error loading native library....Value cannot be null. (Parameter 'provider')"

a5rar commented 1 year ago

Im experiencing this same issue on a maven project. Is there a workaround

finderAUT commented 1 year ago

@a5rar same workaround as above: just copy libgrpc_csharp_ext.arm64.dylib to the directory given in the error message (should be something like /Users/${USER}/.azure-functions-core-tools/Functions/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle.Preview/*/bin)

To automate that for other Devs you can combine Maven OS profiles and the exec-plugin

jasonbarden commented 1 year ago

I'm having this same issue on the new M3 Mac using the workaround provided above.

With 2.46.6 I get the following:

[2023-11-10T15:51:24.476Z] A host error has occurred during startup operation '[...]'. [2023-11-10T15:51:24.477Z] Grpc.Core: Error loading native library "[...]/bin/Debug/net7.0/.azurefunctions/runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib".

With 2.41.0 I get:

[2023-11-10T15:54:12.092Z] A host error has occurred during startup operation '[...]'. [2023-11-10T15:54:12.092Z] Microsoft.Azure.WebJobs.Extensions.DurableTask: Could not load type 'Microsoft.Azure.WebJobs.Host.Scale.ITargetScalerProvider' from assembly 'Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

ved-leachim commented 10 months ago

My system: Chip: Apple M1 OS version: Sonoma 14.2.1

I get the following error using 2.46.6 of Contrib.Grpc.Core.M1:

"Grpc.Core: Error loading native library....Value cannot be null. (Parameter 'provider')"

Using 2.41.0 of Contrib.Grpc.Core.M1 works, thanks!

OlofMorra commented 9 months ago

I'm having this same issue on the new M3 Mac using the workaround provided above.

With 2.46.6 I get the following:

[2023-11-10T15:51:24.476Z] A host error has occurred during startup operation '[...]'. [2023-11-10T15:51:24.477Z] Grpc.Core: Error loading native library "[...]/bin/Debug/net7.0/.azurefunctions/runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib".

With 2.41.0 I get:

[2023-11-10T15:54:12.092Z] A host error has occurred during startup operation '[...]'. [2023-11-10T15:54:12.092Z] Microsoft.Azure.WebJobs.Extensions.DurableTask: Could not load type 'Microsoft.Azure.WebJobs.Host.Scale.ITargetScalerProvider' from assembly 'Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

My system: Chip: Apple M3 Pro 18GB OS version: Sonoma 14.2.1

I am having the Grpc.Core: Error loading native library error for any version now on a Mac M3 Pro. Additionally, I tried to copy the dylib to the correct folder myself after build, but then I receive the following error: Grpc.Core: Error loading native library "/Users/olofmorra/RiderProjects/OneBoard/Infrastructure/bin/output/.azurefunctions/libgrpc_csharp_ext.arm64.dylib". .

Do we have any update on this issue? Because of this reason I cannot run Durable Functions locally with .NET 8...

OlofMorra commented 9 months ago

I'm having this same issue on the new M3 Mac using the workaround provided above. With 2.46.6 I get the following: [2023-11-10T15:51:24.476Z] A host error has occurred during startup operation '[...]'. [2023-11-10T15:51:24.477Z] Grpc.Core: Error loading native library "[...]/bin/Debug/net7.0/.azurefunctions/runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib". With 2.41.0 I get: [2023-11-10T15:54:12.092Z] A host error has occurred during startup operation '[...]'. [2023-11-10T15:54:12.092Z] Microsoft.Azure.WebJobs.Extensions.DurableTask: Could not load type 'Microsoft.Azure.WebJobs.Host.Scale.ITargetScalerProvider' from assembly 'Microsoft.Azure.WebJobs.Host, Version=3.0.34.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.

My system: Chip: Apple M3 Pro 18GB OS version: Sonoma 14.2.1

I am having the Grpc.Core: Error loading native library error for any version now on a Mac M3 Pro. Additionally, I tried to copy the dylib to the correct folder myself after build, but then I receive the following error: Grpc.Core: Error loading native library "/Users/olofmorra/RiderProjects/OneBoard/Infrastructure/bin/output/.azurefunctions/libgrpc_csharp_ext.arm64.dylib". .

Do we have any update on this issue? Because of this reason I cannot run Durable Functions locally with .NET 8...

Ah my mistake. Fixed my problems with the explanation given by @LloydNicholson, see https://github.com/microsoft/durabletask-dotnet/issues/148#issuecomment-1729006827

bochogy commented 8 months ago

Any updates whether there is a permanent fix for this already from your team? @jviau thanks!

iamalexmang commented 7 months ago

Confirm, this error practically makes it impossible to develop Azure Functions on macs with Apple Silicon

Yogeswaran66 commented 7 months ago

This solution doesn't work for .NET 8. Can anyone suggest a workaround for .NET 8 ?

OlofMorra commented 7 months ago

@Yogeswaran66 use the solution by @LloydNicholson in comment https://github.com/microsoft/durabletask-dotnet/issues/148#issuecomment-1729006827

Yogeswaran66 commented 7 months ago

@Yogeswaran66 use the solution by @LloydNicholson in comment #148 (comment)

It's not working for .NET 8

OlofMorra commented 7 months ago

@Yogeswaran66 Have you set the version of Contrib.Grpc.Core.M1 to 2.41.0?

Yogeswaran66 commented 7 months ago

@Yogeswaran66 Have you set the version of Contrib.Grpc.Core.M1 to 2.41.0?

Yes

OlofMorra commented 7 months ago

This is my setup, and it works on my laptop:

So in the .csproj file I have something like the following:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <AzureFunctionsVersion>v4</AzureFunctionsVersion>
    <OutputType>Exe</OutputType>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <!-- See for reason why we need this https://github.com/microsoft/durabletask-dotnet/issues/148#issuecomment-1729006827 -->
  <Target Name="CopyGrpcNativeAssetsToOutDir" AfterTargets="Build">
    <ItemGroup>
      <NativeAssetToCopy Condition="$([MSBuild]::IsOSPlatform('OSX'))" Include="$(OutDir)runtimes/osx-arm64/native/*"/>
    </ItemGroup>
    <Copy SourceFiles="@(NativeAssetToCopy)" DestinationFolder="$(OutDir).azurefunctions/runtimes/osx-arm64/native"/>
  </Target>

<ItemGroup>
    <!-- My extra dependencies -->

    <!-- See for reason why we need version 2.41.0 explicitly https://github.com/microsoft/durabletask-dotnet/issues/148#issuecomment-1729006827 -->
    <PackageReference Include="Contrib.Grpc.Core.M1" Version="2.41.0"/>

    <!-- My Azure Functions dependencies -->
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.20.1"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.1.0"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.CosmosDB" Version="4.7.0"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.0"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.0"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.3.0"/>
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.16.4"/>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.CosmosDB" Version="4.5.0" />
  </ItemGroup>
</Project>

And I run dotnet-isolated functions only, I thought the other type is not supported (or will not be in the future). And I have installed azurite via NPM which I run with the command azurite --silent --debug c:\azurite\debug.log --inMemoryPersistence false BEFORE I start the project.

Hope this helps!

bot109493 commented 6 months ago

@davidmrdavid @OlofMorra Chip: M3 I'm using Camunda 8, and Camunda 8 uses the Zb-client nuget package with Grpc as a dependency, but the Zb-Client nuget package does not support Grpc 2.41.0. Is there any workaround for Grpc 2.46.6?

OlofMorra commented 6 months ago

@davidmrdavid @OlofMorra

Chip: M3

I'm using Camunda 8, and Camunda 8 uses the Zb-client nuget package with Grpc as a dependency, but the Zb-Client nuget package does not support Grpc 2.41.0. Is there any workaround for Grpc 2.46.6?

Not that I know of, you could try to downgrade Zb-Client/Camunda 8 perhaps?

bot109493 commented 6 months ago

Yeah, I did that. However, lower versions of Zb-Client do not support 2.41.0. The lower version of zb-client is supporting 2.41.1

davidmrdavid commented 6 months ago

Hi all, apologies that this remains a challenge, it's something I'm looking into.

Something I've noticed is that I'm unable to reproduce this issue in .NET8 - there, I don't seem to need this workaround to get the application to build. However, I do need it in .NET7 - is everyone's understanding that this is somehow .NET7 specific, or is anyone getting this issue w/ .NET8?

That will help me narrow down a solution here.

iamalexmang commented 6 months ago

Occurs for .NET 8 as well

OlofMorra commented 6 months ago

Having the problem for .NET 8 as well, see setup of project in https://github.com/microsoft/durabletask-dotnet/issues/148#issuecomment-2047844215

BoasHoeven commented 6 months ago

Will this be fixed anytime soon? Having to do all these workarounds / spinning up a windows vm every time I work with durable functions does get old.

davidmrdavid commented 6 months ago

Thanks @iamalexmang, @OlofMorra. Somehow we're unable to repro in .NET8 using Mac M1 (at least we were unable to repro last Friday). I'll circle back again to re-test. FYI @lilyjma

I want to fix this promptly, I'm trying to figure out what is the fastest way to provide a permanent solution here. But I'll need to understand why we're having issues repro'ing this in .NET8 first.

OlofMorra commented 6 months ago

@davidmrdavid See my reproduction repository here: https://github.com/OlofMorra/microsoft-durable-task-issue-148

@bot109493 I got the above sample repository to work with Grpc.Core.M1 2.46.7 instead of 2.41.0, so maybe you can update as well!

davidmrdavid commented 5 months ago

@lilyjma: are you able to please try the .NET8-based repro above on your M1 mac?

Bouke commented 5 months ago

@davidmrdavid The suggested package "Contrib.Grpc.Core.M1" is afaik from a non-affiliated third-party. I think it is a bad idea to suggest to people to consume packages from third-parties.

It seems to me the actual fix is to stop using the deprecated and soon unsupported use of grpc.core and switch to grpc-dot: https://github.com/grpc/grpc/issues/35414.

I'm running into this as well with .NET 8. As soon as I inject a DurableTaskClient into an HTTP trigger, the problem appears.

using System.Net;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask.Client;

namespace Company.FunctionApp1;

public class HttpTrigger1
{
    [Function("HttpTrigger1")]
    public HttpResponseData Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req,
+       [DurableClient] DurableTaskClient client,
        FunctionContext executionContext)
    {
        return req.CreateResponse(HttpStatusCode.OK);
    }
}

[2024-06-11T21:18:17.788Z] A host error has occurred during startup operation '8f5e2dc6-a731-4729-a071-d8a08e187625'. [2024-06-11T21:18:17.788Z] Grpc.Core: Error loading native library. Not found in any of the possible locations: /private/tmp/Company.FunctionApp1/bin/Debug/net8.0/.azurefunctions/libgrpc_csharp_ext.arm64.dylib,/private/tmp/Company.FunctionApp1/bin/Debug/net8.0/.azurefunctions/runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib,/private/tmp/Company.FunctionApp1/bin/Debug/net8.0/.azurefunctions/../../runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib.

Using Contrib.Grpc.Core.M1 version 2.46.7 with the <Target Name="CopyGrpcNativeAssetsToOutDir"... seems to work (doesn't crash on start).

lilyjma commented 5 months ago

Hi all, for some reason, I haven't been able to reproduce this on my M1 mac (Apple M1 Pro, version 14.5).

I cloned the repo @OlofMorra provided here, installed the .NET version mentioned and the mono-libgdiplus package. Then ran the app. Everything worked as expected.

The only difference is the editor - I used Visual Studio Code, so unsure if Jetbrains Rider is making a difference here.

I have Functions Core Tool version 4.0.5801.

OlofMorra commented 5 months ago

@lilyjma I have tried to run it via Visual Studio Code, but it did not work.

It looks like the problem is not related to durabletask-dotnet, but to azure-functions-core-tools. More specifically, azure-functions-core-tools depends on Grpc.Core, which does not have a runtime lib for osx-arm64.

The actual problem is fairly clear to me now, after building your project a dylib file is missing in bin/output/.azurefunctions/runtimes/, namely the folder osx-arm64 with corresponding lib file, which is necessary for Grpc.Core. This will not be solved by Grpc.Core (see https://github.com/grpc/grpc/issues/36836), so the fix that is provided in this thread works, or the fix by installing Rosetta in https://github.com/grpc/grpc/issues/36836 should work (did not work on my M3 though by just installing Rosetta 2 via the terminal, no extra time invested in debugging or anything).

paulverbeke commented 2 months ago

Hi, I've this problem also on mac M1, dotnet 8 isolated, using Jetbrains Rider 2024.2.4 with Azure Toolkit plugin 4.1.3 (Functions Core Tool version 4.0.6280). Working fine on another mac M3 same project same setup (same Rider version, Azure Toolkit plugin etc) 🤷‍♂️

davidmrdavid commented 1 month ago

Part of our plan is to migrate out of the current GRPC provider we're using, into the one provided by AspNetCore, which we understand works more reliably and is the same as what the Azure Functions Host/runtime uses. This is currently gated behind the release of Durable Functions WebJobs extension v3 (which will include the new AspNetCore GRPC dependency, as well as updating other transitive dependencies), and after that we'll need to make a release where we swap our GRPC provider (DF v3's first release will only include the new dependency, then we'll need to actually invoke it over the current GRPC implementation)