search

Particular / ServiceControl

Backend for ServiceInsight and ServicePulse
https://docs.particular.net/servicecontrol/
Other
51 stars 47 forks source link

DependencyResolutionException reported in logs of ServiceControl Monitoring 4.19.0 #2593

Closed JSCProjects closed 3 years ago

JSCProjects commented 3 years ago

ServiceControl Monitoring 4.19.0 throws Autofac.Core.DependencyResolutionException when it receives a message from an endpoint using an unsupported version of NServiceBus.Metrics.ServiceControl.

Who's affected

Users running ServiceControl Monitoring 4.19.0 with endpoints that use NServiceBus.Metrics.ServiceControl version 2.0.x.

Symptoms

After upgrading to ServiceControl Monitoring 4.19.0, the ServiceControl Monitoring logs are flooded with instances of the following exception:

Autofac.Core.DependencyResolutionException: An exception was thrown while activating ServiceControl.Monitoring.QueueLength.LegacyQueueLengthReportHandler. ---> Autofac.Core.DependencyResolutionException: None of the constructors found with 'Autofac.Core.Activators.Reflection.DefaultConstructorFinder' on type 'ServiceControl.Monitoring.QueueLength.LegacyQueueLengthReportHandler' can be invoked with the available services and parameters:
Cannot resolve parameter 'LegacyQueueLengthEndpoints legacyEndpoints' of constructor 'Void .ctor(LegacyQueueLengthEndpoints)'.
   at Autofac.Core.Activators.Reflection.ReflectionActivator.GetValidConstructorBindings(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
   --- End of inner exception stack trace ---
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
   at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
   at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType)
   at NServiceBus.Extensions.Hosting.ServiceProviderAdapter.ChildScopeAdapter.Build(Type typeToBuild) in /_/src/NServiceBus.Extensions.Hosting/ServiceProviderAdapter.cs:line 66
   at NServiceBus.LoadHandlersConnector.<Invoke>d__1.MoveNext() in /_/src/NServiceBus.Core/Pipeline/Incoming/LoadHandlersConnector.cs:line 45
...

Notes

Prior to upgrading to ServiceControl Monitoring 4.19.0, the ServiceControl Monitoring logs will contain messages like this once per hour:

Legacy queue length report received from {endpointInstanceId.InstanceName} instance of {endpointInstanceId.EndpointName}

These endpoints are using an outdated and unsupported version of the NServiceBus.Metrics.ServiceControl package which should be upgraded to the latest supported versions. This will prevent the ServiceControl Monitoring instance from throwing the listed exception.

Original issue ## After installing v4.19.0 I receive alot of exception After installing v.4.19.0 I receive on some service control monitoring instances the following error NServiceBus.Metrics.MetricReport ``` Autofac.Core.DependencyResolutionException: An exception was thrown while activating ServiceControl.Monitoring.QueueLength.LegacyQueueLengthReportHandler. ---> Autofac.Core.DependencyResolutionException: None of the constructors found with 'Autofac.Core.Activators.Reflection.DefaultConstructorFinder' on type 'ServiceControl.Monitoring.QueueLength.LegacyQueueLengthReportHandler' can be invoked with the available services and parameters: Cannot resolve parameter 'LegacyQueueLengthEndpoints legacyEndpoints' of constructor 'Void .ctor(LegacyQueueLengthEndpoints)'. at Autofac.Core.Activators.Reflection.ReflectionActivator.GetValidConstructorBindings(IComponentContext context, IEnumerable`1 parameters) at Autofac.Core.Activators.Reflection.ReflectionActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters) at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget) --- End of inner exception stack trace --- at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget) at Autofac.Core.Lifetime.LifetimeScope.GetOrCreateAndShare(Guid id, Func`1 creator) at Autofac.Core.Resolving.InstanceLookup.Execute() at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters) at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters) at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance) at Autofac.ResolutionExtensions.ResolveOptionalService(IComponentContext context, Service service, IEnumerable`1 parameters) at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetService(Type serviceType) at NServiceBus.Extensions.Hosting.ServiceProviderAdapter.ChildScopeAdapter.Build(Type typeToBuild) in /_/src/NServiceBus.Extensions.Hosting/ServiceProviderAdapter.cs:line 66 at NServiceBus.LoadHandlersConnector.d__1.MoveNext() in /_/src/NServiceBus.Core/Pipeline/Incoming/LoadHandlersConnector.cs:line 45 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.ScheduledTaskHandlingBehavior.d__1.MoveNext() in /_/src/NServiceBus.Core/Scheduling/ScheduledTaskHandlingBehavior.cs:line 22 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at ServiceControl.Monitoring.MessagePoolReleasingBehavior.d__0.MoveNext() in /_/src/ServiceControl.Monitoring/Bootstrapper.cs:line 164 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at NServiceBus.DeserializeMessageConnector.d__1.MoveNext() in /_/src/NServiceBus.Core/Pipeline/Incoming/DeserializeMessageConnector.cs:line 33 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.ProcessingStatisticsBehavior.d__0.MoveNext() in /_/src/NServiceBus.Core/Performance/Statistics/ProcessingStatisticsBehavior.cs:line 25 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.TransportReceiveToPhysicalMessageConnector.d__1.MoveNext() in /_/src/NServiceBus.Core/Pipeline/Incoming/TransportReceiveToPhysicalMessageConnector.cs:line 39 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.MainPipelineExecutor.d__1.MoveNext() in /_/src/NServiceBus.Core/Pipeline/MainPipelineExecutor.cs:line 45 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.Transport.AzureServiceBus.MessagePump.d__23.MoveNext() ```
fabianPas commented 3 years ago

+1 we are at over 100k exceptions currently

mikeminutillo commented 3 years ago

@JSCProjects @fabianPas We are looking into this now. We did some restructuring to the internal DI configuration in 4.19 and it looks like we missed an edge case.

In this case, the handler that is throwing the exception is only there to log a warning (one per hour) when it detects an endpoint using an older method to send metric data. It looks like we don't have a test for that because all it does is log.

If you look at your logs from before the upgrade, please look for a log message that looks like this:

Legacy queue length report received from {endpointInstanceId.InstanceName} instance of {endpointInstanceId.EndpointName}

If you find one, can you let me know which versions of NServiceBus.Metrics and NServiceBus.Metrics.ServiceControl they are using. They should be updated to newer versions because we don't use any of the data from the message they are sending except to record that the endpoint is still alive.

In the meantime I will get a PR out for a fix.

Sorry for the trouble.

JSCProjects commented 3 years ago

Hi @mikeminutillo thanx for the update I found 3 endpoints who are using the following versions

  <package id="NServiceBus.Metrics" version="2.0.0" targetFramework="net452" />
  <package id="NServiceBus.Metrics.ServiceControl" version="2.0.0" targetFramework="net452" />
mikeminutillo commented 3 years ago

We have a solution #2594 which needs to go through review and smoke testing before it is released. I'll let you know when that happens.

The package that is sending the older message type is NServiceBus.Metrics.ServiceControl version 2.0.0. If these endpoints are updated to use NServiceBus.Metrics.ServiceControl 2.1.x then they will stop sending this message and the ServiceControl Monitoring instance will stop throwing the exception.

mikeminutillo commented 3 years ago

@JSCProjects @fabianPas the fix for this has been merged and will be released as part of ServiceControl 4.20. In the meantime, you should go ahead and update the endpoints which are using NServiceBus.Metrics.ServiceControl 2.0.x as this version is unsupported. Doing this will resolve the issue even before ServiceControl 4.20 is released.

mikeminutillo commented 3 years ago

Hi all,

ServiceControl 4.20 has been released. Please upgrade as soon as you are able and let us know if you continue to experience issues.

Thanks for bringing this to our attention, and for your patience while we got it fixed.

-Mike

SzymonPobiega commented 3 years ago

@JSCProjects @fabianPas the fix has been released in ServiceControl 4.20.0