Open jviau opened 1 month ago
Vote up. We have a lot of RPC exception when send a bulk of messages to Azure message bus. When we did manual testing ( a few message we don't have this issue). When we did Load test with around 6000 messages. We got 9.50k Rpc exception as below:
Exception while executing function: Functions.[xxx] Result: Failure
Exception: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Azure.Functions.Worker.Extensions.Abstractions, Version=1.3.0.0, Culture=neutral, PublicKeyToken=551316b6919f366c'. The system cannot find the file specified.
File name: 'Microsoft.Azure.Functions.Worker.Extensions.Abstractions, Version=1.3.0.0, Culture=neutral, PublicKeyToken=551316b6919f366c'
at System.ModuleHandle.ResolveType(QCallModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(MetadataToken caCtorToken, MetadataImport& scope, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1& derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctorWithParameters, Boolean& isVarArg)
at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType)
at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeParameterInfo parameter, RuntimeType caType)
at System.Reflection.RuntimeParameterInfo.GetDefaultValueFromCustomAttributes()
at System.Reflection.RuntimeParameterInfo.TryGetDefaultValueInternal(Boolean raw, Object& defaultValue)
at System.Reflection.RuntimeParameterInfo.get_HasDefaultValue()
at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition.<.ctor>b__2_7(ParameterInfo p) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 59
at System.Linq.Enumerable.WhereSelectArrayIterator`2.ToArray()
at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1 items)
at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition..ctor(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 56
at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToFunctionDefinition(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 118
at Microsoft.Azure.Functions.Worker.GrpcWorker.FunctionLoadRequestHandler(FunctionLoadRequest request, IFunctionsApplication application, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 268
Stack: at System.ModuleHandle.ResolveType(QCallModule module, Int32 typeToken, IntPtr* typeInstArgs, Int32 typeInstCount, IntPtr* methodInstArgs, Int32 methodInstCount, ObjectHandleOnStack type)
at System.ModuleHandle.ResolveTypeHandle(Int32 typeToken, RuntimeTypeHandle[] typeInstantiationContext, RuntimeTypeHandle[] methodInstantiationContext)
at System.Reflection.RuntimeModule.ResolveType(Int32 metadataToken, Type[] genericTypeArguments, Type[] genericMethodArguments)
at System.Reflection.CustomAttribute.FilterCustomAttributeRecord(MetadataToken caCtorToken, MetadataImport& scope, RuntimeModule decoratedModule, MetadataToken decoratedToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1& derivedAttributes, RuntimeType& attributeType, IRuntimeMethodInfo& ctorWithParameters, Boolean& isVarArg)
at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes)
at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType)
at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeParameterInfo parameter, RuntimeType caType)
at System.Reflection.RuntimeParameterInfo.GetDefaultValueFromCustomAttributes()
at System.Reflection.RuntimeParameterInfo.TryGetDefaultValueInternal(Boolean raw, Object& defaultValue)
at System.Reflection.RuntimeParameterInfo.get_HasDefaultValue()
at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition.<.ctor>b__2_7(ParameterInfo p) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 59
at System.Linq.Enumerable.WhereSelectArrayIterator`2.ToArray()
at System.Collections.Immutable.ImmutableArray.CreateRange[T](IEnumerable`1 items)
at Microsoft.Azure.Functions.Worker.Definition.GrpcFunctionDefinition..ctor(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Definition\GrpcFunctionDefinition.cs:line 56
at Microsoft.Azure.Functions.Worker.Rpc.RpcExtensions.ToFunctionDefinition(FunctionLoadRequest loadRequest, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\RpcExtensions.cs:line 118
at Microsoft.Azure.Functions.Worker.GrpcWorker.FunctionLoadRequestHandler(FunctionLoadRequest request, IFunctionsApplication application, IMethodInfoLocator methodInfoLocator) in D:\a\_work\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 268
@goldmelodyvn that call stack is not related to this issue. Please open a separate issue.
@jviau Referring to this issue (https://github.com/Azure/azure-functions-host/issues/10251), may I know if there is an ETA for rollout? My host function version is 4.36 currently. Also, I would like to know any workaround is available?
An update on the roll out is highly appreciated. We are stuck now and need to resolve ASAP. Cheers
The current design of the extension RPC server has it piggy backing off the general worker gRPC server. The lifetime mismatch between individual workers and this server causes issues during job host transitions. We will need explore one of two options: