search

Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.94k stars 442 forks source link

Incorrect version of Grpc is being used (package version ignored) #4527

Open sdebruyn opened 5 years ago

sdebruyn commented 5 years ago

I am using the NuGet package Google.Cloud.Firestore which relies on some other packages where type forwarding is used. The forwarded types are not found and result in a MissingMethodException.

The issue occurs when debugging locally in VS Code on macOS. I haven't deployed this yet.

The same code works in a .NET Core console app.

A lot more details at the original issue: https://github.com/googleapis/google-cloud-dotnet/issues/3101

Investigative information

Repro steps

See original issue above

Expected behavior

Type forwarding is working and no MissingMethodException

Actual behavior

MissingMethodException when this calls into a forwarded type.

Known workarounds

?

Related information

Csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <AzureFunctionsVersion>v2</AzureFunctionsVersion>
    <RootNamespace>myappname</RootNamespace>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.4"/>
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24"/>
    <PackageReference Include="PuppeteerSharp" Version="1.17.1"/>
    <PackageReference Include="Google.Cloud.Firestore" Version="1.0.0-beta20"/>
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="google.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>
jskeet commented 5 years ago

Note: the same issue occurs using the emulator from Windows and VS2019.

jskeet commented 5 years ago

I have a simpler repro now - there's no need for the Firestore dependency.

Project file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <AzureFunctionsVersion>v2</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Grpc.Auth" Version="1.21.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

Function source:

using Google.Apis.Auth.OAuth2;
using Grpc.Auth;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using System.Threading;
using System.Threading.Tasks;

namespace FunctionApp1
{
    public static class Function1
    {
        [FunctionName("Function1")]
        public static IActionResult Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            var tokenAccess = new DummyTokenAccess();
            // This succeeds
            tokenAccess.ToCallCredentials();
            // This fails
            tokenAccess.ToChannelCredentials();

            return new OkObjectResult("Passed");
        }
    }

    class DummyTokenAccess : ITokenAccess
    {
        public Task<string> GetAccessTokenForRequestAsync(string authUri = null, CancellationToken cancellationToken = default)
        {
            throw new System.NotImplementedException();
        }
    }
}

Exception:

System.MissingMethodException
  HResult=0x80131513
  Message=Method not found: 'Grpc.Core.ChannelCredentials Grpc.Core.ChannelCredentials.Create(Grpc.Core.ChannelCredentials, Grpc.Core.CallCredentials)'.
  Source=Grpc.Auth
  StackTrace:
   at Grpc.Auth.GoogleGrpcCredentials.ToChannelCredentials(ITokenAccess googleCredential)
   at FunctionApp1.Function1.Run(HttpRequest req, ILogger log) in C:\Users\skeet\Test\Projects\Issues\google-cloud-dotnet\Issue3101\Issue3101\FunctionApp1\Function1.cs:line 26

This is very odd indeed:

// Declaration in Grpc.Core, in ChannelCredentials:
.method public hidebysig static class Grpc.Core.ChannelCredentials 
        Create(class Grpc.Core.ChannelCredentials channelCredentials,
               class [Grpc.Core.Api]Grpc.Core.CallCredentials callCredentials) cil managed

// Call in Grpc.Auth, in GoogleGrpcCredentials.ToChannelCredentials:
call       class [Grpc.Core]Grpc.Core.ChannelCredentials [Grpc.Core]Grpc.Core.ChannelCredentials::Create(class [Grpc.Core]Grpc.Core.ChannelCredentials,
                                                                                                                     class [Grpc.Core.Api]Grpc.Core.CallCredentials)

Note that the ToCallCredentials method works. CallCredentials is in Grpc.Core.Api; ChannelCredentials is in Grpc.Core.

I'm going to keep investigating this a bit to see if I can reproduce this without even a Grpc.Auth reference...

jskeet commented 5 years ago

Okay, even simpler repro - and an explanation.

First, the project file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <AzureFunctionsVersion>v2</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Grpc.Core" Version="1.21.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

Next, source code:

using Grpc.Core;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;

namespace FunctionApp1
{
    public static class Function1
    {
        [FunctionName("Function1")]
        public static IActionResult Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req)
        {
            CreateChannelCredentials(null);
            return new OkObjectResult("Passed");
        }

        private static ChannelCredentials CreateChannelCredentials(CallCredentials callCredentials) =>
            ChannelCredentials.Create(new SslCredentials(), callCredentials);
    }
}

This fails with the same MissingMethodException when you try to execute CreateChannelCredentials, when the JIT compiler works on that method - because it can't find the method.

The problem is that it's not loading Grpc.Core 1.21.0 at all - it's using 1.18.0 which is in C:\Users\skeet\AppData\Local\AzureFunctionsTools\Releases\2.22.0\cli\

If you update the cli directory to contain the relevant files from Grpc.Core and Grpc.Core.Api, and update func.deps.json accordingly, it all works - including Google.Cloud.Firestore.

So basically, this is a problem for any function needing to use a version of Grpc.Core later than the one embedded in the CLI application.

AartBluestoke commented 5 years ago

Behind the scenes the following sequence occurs. . You compile and include the azure functions sdk, vs finds that it you load 1.21 all the binary dependencies can be satisfied, so complies without error. . The project starts to run . Azure functions host starts, and loads 1.18 . Azure functions loads your functions .net runtime notes that the library is present in the app domain and so can't load another copy. . At runtime you try to use a 1.21 feature and exception as noted.

This is because the functions runtime in azure loads that specific version of the library , but the announced dependencies are often vague. If you use features from a library newer that what is present things fail in the way described. The same goes of you assume bugs are fixed. This happens for other libraries as well, e.g. newtonsoft.json https://github.com/Azure/azure-functions-host/issues/4049#issuecomment-505133197 For the json dependency they fixed this by causing the functions sdk to depend on the exact version loaded by the runtime, but this block including in your prophecy any library that includes any other version. Probably a more helpful failure mechanism imo).

fabiocav commented 5 years ago

Will investigate this as the version packaged with the CLI shouldn't cause this issue, so looks like there's a bug here. Do you happen to know it this is happening outside of the context of the CLI?

jskeet commented 5 years ago

@fabiocav: I don't know, I'm afraid. I assume that VS just launches the CLI? How can we test this outside the context of the CLI?

fabiocav commented 5 years ago

This was just in case you had tested this on Azure or in one of the container options. Wasn't really expecting that to be the case, but thought I'd ask.

I'll assign this to the next sprint and take a closer look to see what is causing the issue.

andreujuanc commented 5 years ago

@fabiocav I just had this problem and when @jskeet wrote "func.deps.json" it opened my eyes and I just remembered this issue: https://github.com/Azure/azure-functions-host/issues/4304

So I just added your fix for the WCF ref to my project file and boom, it worked:

<Target Name="PostBuild" AfterTargets="PostBuildEvent">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <!--<Exec Command="copy $(OutDir)bin\runtimes\win\lib\netstandard2.0\System.Private.ServiceModel.dll $(OutDir)bin\System.Private.ServiceModel.dll" />-->
    <Exec Command="copy $(OutDir)$(ProjectName).deps.json $(OutDir)bin\function.deps.json" />
  </Target>
  <Target Name="PostPublish" BeforeTargets="AfterPublish">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <!--<Exec Command="copy $(PublishDir)bin\runtimes\win\lib\netstandard2.0\System.Private.ServiceModel.dll $(PublishDir)bin\System.Private.ServiceModel.dll" />-->
    <Exec Command="copy $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin\function.deps.json" />
  </Target>
KasunKoswattha commented 5 years ago

I'm trying to make this work on a mac. Is there a workaround or a solution ?

jslaybaugh commented 5 years ago

@KasunKoswattha On a mac, the copy command doesn't exist, so I've gotten past this issue by using 

<Target Name="PostBuild" AfterTargets="PostBuildEvent">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <!--<Exec Command="copy $(OutDir)bin\runtimes\win\lib\netstandard2.0\System.Private.ServiceModel.dll $(OutDir)bin\System.Private.ServiceModel.dll" />-->
    <Exec Command="cp $(OutDir)$(ProjectName).deps.json $(OutDir)bin\function.deps.json" />
  </Target>
  <Target Name="PostPublish" BeforeTargets="AfterPublish">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <!--<Exec Command="copy $(PublishDir)bin\runtimes\win\lib\netstandard2.0\System.Private.ServiceModel.dll $(PublishDir)bin\System.Private.ServiceModel.dll" />-->
    <Exec Command="cp $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin\function.deps.json" />
  </Target>

or in my specific case, where we need to run Windows AND OSX:

<Target Name="PostBuildWindows" AfterTargets="PostBuildEvent" Condition="'$(OS)' == 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Exec Command="copy $(OutDir)$(ProjectName).deps.json $(OutDir)bin\function.deps.json" />
    <Message Text="PostBuild Copied $(OutDir)bin\function.deps.json" />  
  </Target>
  <Target Name="PostPublishWindows" BeforeTargets="AfterPublish" Condition="'$(OS)' == 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Message Text="PostPublish Copied $(OutDir)bin\function.deps.json" />  
    <Exec Command="copy $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin\function.deps.json" />
  </Target>
  <Target Name="PostBuildUnix" AfterTargets="PostBuildEvent" Condition="'$(OS)' != 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Exec Command="cp -r $(OutDir)$(ProjectName).deps.json $(OutDir)bin/function.deps.json" />
    <Message Text="PostBuild Copied $(OutDir)bin/function.deps.json" />  
  </Target>
  <Target Name="PostPublishUnix" BeforeTargets="AfterPublish" Condition="'$(OS)' != 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Exec Command="cp -r $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin/function.deps.json" />
    <Message Text="PostPublish Copied $(OutDir)bin/function.deps.json" />  
  </Target>

However, I now have the following exception I can't seem to solve in my Azure function on OSX:

System.Private.CoreLib: Exception while executing function: CleanupQueue. Grpc.Core: The native method "grpcsharp_batch_context_recv_message_next_slice_peek" does not exist.
System.MissingMethodException: The native method "grpcsharp_batch_context_recv_message_next_slice_peek" does not exist
   at Grpc.Core.Internal.UnmanagedLibrary.GetNativeMethodDelegate[T](String methodName)
   at Grpc.Core.Internal.NativeMethods..ctor(UnmanagedLibrary library)
   at Grpc.Core.Internal.NativeExtension.LoadNativeMethods()
   at Grpc.Core.Internal.NativeExtension..ctor()
   at Grpc.Core.Internal.NativeExtension.Get()
   at Grpc.Core.GrpcEnvironment.GrpcNativeInit()
   at Grpc.Core.GrpcEnvironment..ctor()
   at Grpc.Core.GrpcEnvironment.AddRef()
   at Grpc.Core.Channel..ctor(String target, ChannelCredentials credentials, IEnumerable`1 options)
   at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint, IEnumerable`1 channelOptions, ChannelCredentials credentials)
   at Google.Cloud.Firestore.FirestoreDb.Create(String projectId, FirestoreClient client)
   at Utilities.FirebaseUtility.GetEntryQueueStatus() in /Users/jslaybaugh/Projects/brushfire-netcore/src/Utilities/FirebaseUtility.cs:line 25

Can anyone help me figure out what this means? @jskeet ?

jskeet commented 5 years ago

@jslaybaugh: I'm afraid I can't easily tell what your context is, but I suspect it's the same underlying problem... I'm afraid I have no way of trying to support Azure Functions on a Mac.

KasunKoswattha commented 5 years ago

@KasunKoswattha On a mac, the copy command doesn't exist, so I've gotten past this issue by using

<Target Name="PostBuild" AfterTargets="PostBuildEvent">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <!--<Exec Command="copy $(OutDir)bin\runtimes\win\lib\netstandard2.0\System.Private.ServiceModel.dll $(OutDir)bin\System.Private.ServiceModel.dll" />-->
    <Exec Command="cp $(OutDir)$(ProjectName).deps.json $(OutDir)bin\function.deps.json" />
  </Target>
  <Target Name="PostPublish" BeforeTargets="AfterPublish">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <!--<Exec Command="copy $(PublishDir)bin\runtimes\win\lib\netstandard2.0\System.Private.ServiceModel.dll $(PublishDir)bin\System.Private.ServiceModel.dll" />-->
    <Exec Command="cp $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin\function.deps.json" />
  </Target>

or in my specific case, where we need to run Windows AND OSX:

<Target Name="PostBuildWindows" AfterTargets="PostBuildEvent" Condition="'$(OS)' == 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Exec Command="copy $(OutDir)$(ProjectName).deps.json $(OutDir)bin\function.deps.json" />
    <Message Text="PostBuild Copied $(OutDir)bin\function.deps.json" />  
  </Target>
  <Target Name="PostPublishWindows" BeforeTargets="AfterPublish" Condition="'$(OS)' == 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Message Text="PostPublish Copied $(OutDir)bin\function.deps.json" />  
    <Exec Command="copy $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin\function.deps.json" />
  </Target>
  <Target Name="PostBuildUnix" AfterTargets="PostBuildEvent" Condition="'$(OS)' != 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Exec Command="cp -r $(OutDir)$(ProjectName).deps.json $(OutDir)bin/function.deps.json" />
    <Message Text="PostBuild Copied $(OutDir)bin/function.deps.json" />  
  </Target>
  <Target Name="PostPublishUnix" BeforeTargets="AfterPublish" Condition="'$(OS)' != 'Windows_NT'">
    <!--https://github.com/Azure/azure-functions-host/issues/3568-->
    <Exec Command="cp -r $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin/function.deps.json" />
    <Message Text="PostPublish Copied $(OutDir)bin/function.deps.json" />  
  </Target>

However, I now have the following exception I can't seem to solve in my Azure function on OSX:

System.Private.CoreLib: Exception while executing function: CleanupQueue. Grpc.Core: The native method "grpcsharp_batch_context_recv_message_next_slice_peek" does not exist.
System.MissingMethodException: The native method "grpcsharp_batch_context_recv_message_next_slice_peek" does not exist
   at Grpc.Core.Internal.UnmanagedLibrary.GetNativeMethodDelegate[T](String methodName)
   at Grpc.Core.Internal.NativeMethods..ctor(UnmanagedLibrary library)
   at Grpc.Core.Internal.NativeExtension.LoadNativeMethods()
   at Grpc.Core.Internal.NativeExtension..ctor()
   at Grpc.Core.Internal.NativeExtension.Get()
   at Grpc.Core.GrpcEnvironment.GrpcNativeInit()
   at Grpc.Core.GrpcEnvironment..ctor()
   at Grpc.Core.GrpcEnvironment.AddRef()
   at Grpc.Core.Channel..ctor(String target, ChannelCredentials credentials, IEnumerable`1 options)
   at Google.Api.Gax.Grpc.ChannelPool.GetChannel(ServiceEndpoint endpoint, IEnumerable`1 channelOptions, ChannelCredentials credentials)
   at Google.Cloud.Firestore.FirestoreDb.Create(String projectId, FirestoreClient client)
   at Utilities.FirebaseUtility.GetEntryQueueStatus() in /Users/jslaybaugh/Projects/brushfire-netcore/src/Utilities/FirebaseUtility.cs:line 25

Can anyone help me figure out what this means? @jskeet ?

Thanks @jslaybaugh I will try this out. But we really need a solution for this. :(

DJ4ddi commented 5 years ago

I ran into the issue described on StackOverflow, leading me here.

Locally, both manually including a more recent package version of Grpc.Core via NuGet and the workaround posted by @andreujuanc worked. Unfortunately, neither did the trick for the remote end in the Azure cloud. Copying the dependency JSON changes nothing, while upgrading the package changes the original "Could not load type" error to Method not found: 'Grpc.Core.ChannelCredentials Grpc.Auth.GoogleGrpcCredentials.ToChannelCredentials(Google.Apis.Auth.OAuth2.ITokenAccess)'. So, while it does make a difference, I still can't run the function properly. Interestingly, also manually including Grpc.Core.Api restores the original error Could not load type 'Grpc.Core.CallCredentials' from assembly 'Grpc.Core.Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d754f35622e28bad'. And in some combination of workarounds, I also managed to kill the library entirely, as indicated by the fact that grpc_csharp_ext.x86.dll failed to load.

Needless to say, I'm a bit confused. Why does basically every combination of proposed solutions work locally, but none of them solve the problem for the cloud runtime (which, according to some reports, shouldn't even be affected in the first place)?

jskeet commented 5 years ago

@DJ4ddi: If you've upgraded to Grpc.Core version 2.x, that would definitely cause problems for anything expecting 1.x, including (currently) all the Google Cloud client libraries. (These will soon be upgraded to Grpc.Core 2.x - at which point you won't be able to use them in the same app with anything 1.x-based, of course.)

DJ4ddi commented 5 years ago

The symptoms were the exact same when manually installing version 1.22.1 or 1.22.0 (the exact version referenced by Google.Cloud.Dialogflow.V2).

wikes82 commented 5 years ago

the workaround of copying *.deps.json work on local dev machine, but I'm still getting the error on publish to Azure Functions.

jslaybaugh commented 5 years ago

@wikes82 me too! But only if it runs through the publish on Azure DevOps. I've got a ticket open with them now about that. If I right click and tell it to do a zip deploy from VS (Windows) it works in Azure Functions for me with the following code:

    <Target Name="PostBuildWindows" AfterTargets="PostBuildEvent">
        <!-- https://github.com/Azure/azure-functions-host/issues/3568 -->
        <Exec Command="copy $(OutDir)$(ProjectName).deps.json $(OutDir)bin\function.deps.json" />
    </Target>
    <Target Name="PostPublishWindows" BeforeTargets="CreateZipFile">
        <!-- https://github.com/Azure/azure-functions-host/issues/3568 -->
        <Exec Command="copy $(PublishDir)$(ProjectName).deps.json $(PublishDir)bin\function.deps.json" />
    </Target>

Note the BeforeTargets="CreateZipFile" I'm now using.

wikes82 commented 5 years ago

@jslaybaugh I can't find option to do zip deploy on my VS2019 Community

EDIT: NVM, I figured it out

wikes82 commented 5 years ago

I'm getting the following error now Error loading native library. Not found in any of the possible locations: D:\home\site\wwwroot\bin\grpc_csharp_ext.x86.dll,D:\home\site\wwwroot\bin\runtimes/win/native\grpc_csharp_ext.x86.dll,D:\home\site\wwwroot\bin\../..\runtimes/win/native\grpc_csharp_ext.x86.dll

I checked and the file grpc_csharp_ext.x86.dll is in /runtimes/win/native

wikes82 commented 5 years ago

I fixed it by adding this line into @jslaybaugh BeforeTargets="CreateZipFile" <Exec Command="copy $(PublishDir)runtimes\win\native\*.* $(PublishDir)bin\" />

KasunKoswattha commented 4 years ago

Are we going to get a fix for this from Microsoft ?

fabiocav commented 4 years ago

This should be resolved. The fix was pushed a bit ago in version 1.0.30-beta2 of the SDK, but I just pushed an update with some enhancements. Can you please update your reference to 1.0.30-beta3 and retry?

KasunKoswattha commented 4 years ago

This should be resolved. The fix was pushed a bit ago in version 1.0.30-beta2 of the SDK, but I just pushed an update with some enhancements. Can you please update your reference to 1.0.30-beta3 and retry?

Thanks @fabiocav . Let me try it out and get back to you

KasunKoswattha commented 4 years ago

@fabiocav I can confirm that I don't see the issue with 1.0.30-beta. I'm using it with .netcoreapp 2.1 on a mac. Thanks for helping on this. @jskeet Thank you for troubleshooting and providing information.

Zastai commented 4 years ago

Not related to the main issue, but is there any reason people are using <Exec> with copy/cp instead of <Copy> in their workarounds? Seems like a step backwards to start executing system utilities when MSBuild has built-in support.

EthanSK commented 3 years ago

This is still a problem...