Particular / NServiceBus

Build, version, and monitor better microservices with the most powerful service platform for .NET
https://particular.net/nservicebus/
Other
2.1k stars 647 forks source link

NServiceBus.MessageDeserializationException: An error occurred while attempting to extract logical messages from transport message #5144

Closed JSCProjects closed 6 years ago

JSCProjects commented 6 years ago

Hello,

I'm receiving the above error after I've moved some handlers to an dynamically loaded assembly(addin functionality) before the error came the same handler was directly referenced in host. Now I've disconnected this handler from host and let nservicebus load it from a subdirectory and used the AssemblyScanner ScanAssembliesInNestedDirectories = true; This issue looks a lot like #4567 Also note that the messages are using the DataBus!

Below is my endpointConfiguration.

configuration.SendHeartbeatTo("particular.servicecontrol");
configuration.EnableInstallers();

configuration.ConfigureAuditProcessedMessagesTo();
configuration.ConfigureSendFailedMessagesTo();
configuration.ConfigureMetrics();

var recoverabilitySettings = configuration.Recoverability();
recoverabilitySettings.DisableLegacyRetriesSatellite();

var transport = configuration.ToUseAzureServiceBusTransport();
transport.ConnectivityMode(ConnectivityMode.AutoDetect);
transport.MessageReceivers().PrefetchCount(1);
transport.UseForwardingTopology();
configuration.LimitMessageProcessingConcurrencyTo(1);

var sanitization = transport.Sanitization();
sanitization.UseStrategy<ValidateAndHashIfNeeded>();

var persistence = configuration.UsePersistence<AzureStoragePersistence, StorageType.Sagas>();
persistence.ConnectionString(ConnectionStrings.Storage);

configuration.UseSerialization<JsonSerializer>();

var databus = configuration.UseDataBus<AzureDataBus>();
databus.CleanupInterval(0);
databus.ConnectionString(ConnectionStrings.Storage);

configuration.ApplyCustomConventions(events: IsEventNamespace, commands: IsCommandNameSpace);
configuration.ApplyLicenseFromCloudConfigAsync().GetAwaiter().GetResult();

configuration.EnableFeature<Sagas>();

var scanner = configuration.AssemblyScanner();
scanner.ScanAssembliesInNestedDirectories = true;

var builder = new ContainerBuilder();
RegisterTypes(builder);
var container = builder.Build();
configuration.UseContainer<AutofacBuilder>(c => c.ExistingLifetimeScope(container));

I use the following nuget's

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Autofac" version="4.4.0" targetFramework="net452" />
  <package id="DataAnnotationsValidator" version="1.0.0.1" targetFramework="net452" />
  <package id="DeveloperForce.Force" version="1.3.2" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.Client" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNetCore.Http.Abstractions" version="1.1.0" targetFramework="net452" />
  <package id="Microsoft.AspNetCore.Http.Features" version="1.1.0" targetFramework="net452" />
  <package id="Microsoft.Azure.KeyVault.Core" version="2.0.4" targetFramework="net452" />
  <package id="Microsoft.Azure.WebJobs" version="2.0.0" targetFramework="net452" />
  <package id="Microsoft.Azure.WebJobs.Core" version="2.0.0" targetFramework="net452" />
  <package id="Microsoft.CSharp" version="4.4.1" targetFramework="net452" />
  <package id="Microsoft.Data.Edm" version="5.8.3" targetFramework="net452" />
  <package id="Microsoft.Data.OData" version="5.8.3" targetFramework="net452" />
  <package id="Microsoft.Data.Services.Client" version="5.8.3" targetFramework="net452" />
  <package id="Microsoft.Extensions.Primitives" version="1.1.0" targetFramework="net452" />
  <package id="Microsoft.NETCore.Platforms" version="1.1.0" targetFramework="net452" />
  <package id="Microsoft.Web.WebJobs.Publish" version="1.1.0" targetFramework="net452" />
  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="3.2.3" targetFramework="net452" />
  <package id="NETStandard.Library" version="1.6.1" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net452" />
  <package id="NServiceBus" version="6.4.3" targetFramework="net452" />
  <package id="NServiceBus.Autofac" version="6.0.1" targetFramework="net452" />
  <package id="NServiceBus.Azure.Transports.WindowsAzureServiceBus" version="7.2.11" targetFramework="net452" />
  <package id="NServiceBus.DataBus.AzureBlobStorage" version="1.1.0" targetFramework="net452" />
  <package id="NServiceBus.Heartbeat" version="2.0.0" targetFramework="net452" />
  <package id="NServiceBus.Metrics" version="2.0.0" targetFramework="net452" />
  <package id="NServiceBus.Metrics.ServiceControl" version="2.0.1" targetFramework="net452" />
  <package id="NServiceBus.Persistence.AzureStorage" version="1.2.0" targetFramework="net452" />
  <package id="Sendgrid" version="9.1.0" targetFramework="net452" />
  <package id="SSH.NET" version="2016.0.0" targetFramework="net452" />
  <package id="StyleCop.Analyzers" version="1.0.0" targetFramework="net452" developmentDependency="true" />
  <package id="System.Collections" version="4.3.0" targetFramework="net452" />
  <package id="System.Collections.Concurrent" version="4.3.0" targetFramework="net452" />
  <package id="System.ComponentModel.EventBasedAsync" version="4.3.0" targetFramework="net452" />
  <package id="System.Diagnostics.Debug" version="4.3.0" targetFramework="net452" />
  <package id="System.Diagnostics.Tools" version="4.3.0" targetFramework="net452" />
  <package id="System.Diagnostics.Tracing" version="4.3.0" targetFramework="net452" />
  <package id="System.Dynamic.Runtime" version="4.3.0" targetFramework="net452" />
  <package id="System.Globalization" version="4.3.0" targetFramework="net452" />
  <package id="System.IO" version="4.3.0" targetFramework="net452" />
  <package id="System.IO.Compression" version="4.3.0" targetFramework="net452" />
  <package id="System.Linq" version="4.3.0" targetFramework="net452" />
  <package id="System.Linq.Expressions" version="4.3.0" targetFramework="net452" />
  <package id="System.Linq.Queryable" version="4.3.0" targetFramework="net452" />
  <package id="System.Net.Http" version="4.3.1" targetFramework="net452" />
  <package id="System.Net.Primitives" version="4.3.0" targetFramework="net452" />
  <package id="System.Net.Requests" version="4.3.0" targetFramework="net452" />
  <package id="System.ObjectModel" version="4.3.0" targetFramework="net452" />
  <package id="System.Reflection" version="4.3.0" targetFramework="net452" />
  <package id="System.Reflection.Extensions" version="4.3.0" targetFramework="net452" />
  <package id="System.Reflection.Primitives" version="4.3.0" targetFramework="net452" />
  <package id="System.Resources.ResourceManager" version="4.3.0" targetFramework="net452" />
  <package id="System.Runtime" version="4.3.0" targetFramework="net452" />
  <package id="System.Runtime.CompilerServices.Unsafe" version="4.3.0" targetFramework="net452" />
  <package id="System.Runtime.Extensions" version="4.3.0" targetFramework="net452" />
  <package id="System.Runtime.InteropServices" version="4.3.0" targetFramework="net452" />
  <package id="System.Runtime.InteropServices.RuntimeInformation" version="4.3.0" targetFramework="net452" />
  <package id="System.Runtime.Numerics" version="4.3.0" targetFramework="net452" />
  <package id="System.Spatial" version="5.8.3" targetFramework="net452" />
  <package id="System.Text.Encoding" version="4.3.0" targetFramework="net452" />
  <package id="System.Text.Encoding.Extensions" version="4.3.0" targetFramework="net452" />
  <package id="System.Text.Encodings.Web" version="4.3.0" targetFramework="net452" />
  <package id="System.Text.RegularExpressions" version="4.3.0" targetFramework="net452" />
  <package id="System.Threading" version="4.3.0" targetFramework="net452" />
  <package id="System.Threading.Tasks" version="4.3.0" targetFramework="net452" />
  <package id="System.Threading.Timer" version="4.3.0" targetFramework="net452" />
  <package id="System.Xml.ReaderWriter" version="4.3.0" targetFramework="net452" />
  <package id="System.Xml.XDocument" version="4.3.0" targetFramework="net452" />
  <package id="WindowsAzure.ServiceBus" version="4.1.8" targetFramework="net452" />
  <package id="WindowsAzure.Storage" version="8.7.0" targetFramework="net452" />
</packages>
andreasohlund commented 6 years ago

Can you share your message definitions?

Would it be possible to strip the code down to a simple repro that you could share with us? (Since you're on v6.4 you can switch to the learning transport)

JSCProjects commented 6 years ago

Is it possible to sent this private?

andreasohlund commented 6 years ago

Got the repro, will report back here with my findings

andreasohlund commented 6 years ago

Can't see anything wrong with the drop-in assembly, I suspect that there is some references that is causing issues.

Unfortunately the NSB host makes troubleshooting asm scanning issues a bit hard but can you do the following:

  var firstChanceLog = LogManager.GetLogger("FirstChanceException");
            var unhandledLog = LogManager.GetLogger("UnhandledException");
            var domain = AppDomain.CurrentDomain;

            domain.FirstChanceException += (o, ea) => { firstChanceLog.Debug(ea.Exception.Message, ea.Exception); };
            domain.UnhandledException += (o, ea) =>
            {
                var exception = ea.ExceptionObject as Exception;
                if (exception != null) unhandledLog.Error(exception.Message, exception);
            };

Can you try that and start the endpoint with loglevel set to debug?

andreasohlund commented 6 years ago

@JSCProjects did you get a chance to test the above?

JSCProjects commented 6 years ago

Darn, missed that post. Sorry will try now

JSCProjects commented 6 years ago

@andreasohlund Tried it but I can't find any clues

andreasohlund commented 6 years ago

Ok, would it be possible to reduce this to a minimal runnable reproduction that I can try on my end?

JSCProjects commented 6 years ago

@andreasohlund I mailed you a repro

andreasohlund commented 6 years ago

Got it, will see if it repros on my end

JSCProjects commented 6 years ago

Finally found it, if you want to (dynamic) load assemblies you need to load them with Assembly.LoadFrom(path) instead of Assembly.Load(byte[]). Also implement AppDomain.CurrentDomain.AssemblyResolve eventhandler to load deeper references.

@andreasohlund & @dvdstelt thanx for your help

andreasohlund commented 6 years ago

Finally found it, if you want to (dynamic) load assemblies you need to load them with Assembly.LoadFrom(path) instead of Assembly.Load(byte[])

Noting that v7 does use Assembly.LoadFrom

sussexrick commented 3 months ago

In case it helps anyone, I just had the same error message for an event class that had moved namespace. The publishing endpoint had the old code and the receiving endpoint had the new code, hence deserialization failed. Updating the publishing endpoint fixed it.