search

Azure / azure-functions-host

The host/runtime that powers Azure Functions
https://functions.azure.com
MIT License
1.94k stars 441 forks source link

Extensions RPC lifetime mismatch between server and JobHost leads to incorrect RPC calls #10540

Open jviau opened 2 weeks ago

jviau commented 2 weeks ago

The current design of the extension RPC server has it piggy backing off the general worker gRPC server. The lifetime mismatch between individual workers and this server causes issues during job host transitions. We will need explore one of two options:

  1. Improve lifetime tracking with the existing RPC server
    • Only unregister endpoints when functions have finished draining.
    • Support multiple job hosts being registered. Will need to include an identifier for what set of endpoints an extension call needs to be routed to.
  2. OR move these endpoints into a new server that lives in the job host.
    • Will need to measure the impact an extra server startup will have.
goldmelodyvn commented 1 day ago

Vote up. We have a lot of RPC exception when send a bulk of messages to Azure message bus. When we did manual testing ( a few message we don't have this issue). When we did Load test with around 6000 messages. We got 9.50k Rpc exception as below:

Exception while executing function: Functions.[xxx] Result: Failure
Exception: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Azure.Functions.Worker.Extensions.Abstractions, Version=1.3.0.0, Culture=neutral, PublicKeyToken=551316b6919f366c'. The system cannot find the file specified.
File name: 'Microsoft.Azure.Functions.Worker.Extensions.Abstractions, Version=1.3.0.0, Culture=neutral, PublicKeyToken=551316b6919f366c'
   at System.ModuleHandle.ResolveType(QCallModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(MetadataToken caCtorToken, MetadataImport& scope, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1& derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctorWithParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeParameterInfo parameter, RuntimeType caType)
   at System.Reflection.RuntimeParameterInfo.GetDefaultValueFromCustomAttributes()
   at System.Reflection.RuntimeParameterInfo.TryGetDefaultValueInternal(Boolean raw, Object& defaultValue)
   at System.Reflection.RuntimeParameterInfo.get_HasDefaultValue()
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition.<.ctor>b__2_7(ParameterInfo p) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 59
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.ToArray()
   at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1 items)
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition..ctor(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 56
   at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToFunctionDefinition(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 118
   at Microsoft.Azure.Functions.Worker.GrpcWorker.FunctionLoadRequestHandler(FunctionLoadRequest request, IFunctionsApplication application, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 268
Stack:    at System.ModuleHandle.ResolveType(QCallModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
   at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
   at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
   at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(MetadataToken caCtorToken, MetadataImport& scope, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1& derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctorWithParameters, Boolean& isVarArg)
   at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType)
   at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeParameterInfo parameter, RuntimeType caType)
   at System.Reflection.RuntimeParameterInfo.GetDefaultValueFromCustomAttributes()
   at System.Reflection.RuntimeParameterInfo.TryGetDefaultValueInternal(Boolean raw, Object& defaultValue)
   at System.Reflection.RuntimeParameterInfo.get_HasDefaultValue()
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition.<.ctor>b__2_7(ParameterInfo p) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 59
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.ToArray()
   at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1 items)
   at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition..ctor(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 56
   at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToFunctionDefinition(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 118
   at Microsoft.Azure.Functions.Worker.GrpcWorker.FunctionLoadRequestHandler(FunctionLoadRequest request, IFunctionsApplication application, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 268