search

Particular / NServiceBus.DataBus.AzureBlobStorage

DataBus implementation using Azure Blobs to store the payload
https://docs.particular.net/samples/azure/blob-storage-databus/
Other
3 stars 6 forks source link

Incomplete message was send to the queue #22

Closed JSCProjects closed 7 years ago

JSCProjects commented 7 years ago

Hello,

We are sending files as specific commands to specific endpoints. These commands uses the databus functionality for storing the file content. But on one particular moment in time there were messages send to a queue without the databus header, so therefore it was impossible to receive it correct in the receiving handler.

The commands are send via Parallel.ForEach construction and it never failed before. Only for 90 msg's it failed with following exception:

2017-04-02 05:52:38.105 ERROR NServiceBus.Transport.AzureServiceBus.DefaultOutgoingBatchRouter Failed to dispatch a batch with the following message IDs: b6208031-9ed6-4bf9-8ba5-2686abab2fca System.TimeoutException: The request operation did not complete within the allotted timeout of 00:01:10. The time allotted to this operation may have been a portion of a longer timeout. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:edc36893-e81b-4da6-87f7-0662845a8d5c, Timestamp:4/2/2017 3:52:38 AM

Server stack trace:

Exception rethrown at [0]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.DuplexRequestBindingElement.DuplexRequestSessionChannel.EndRequest(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory1.RequestSessionChannel.RequestAsyncResult.<>c.<GetAsyncSteps>b__9_3(RequestAsyncResult thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.StepCallback(IAsyncResult result)

Exception rethrown at [1]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult1.End(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory1.RequestSessionChannel.EndRequest(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory1.RedirectContainerSessionChannel.RequestAsyncResult.<>c__DisplayClass8_1.<GetAsyncSteps>b__4(RequestAsyncResult thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.StepCallback(IAsyncResult result)

Exception rethrown at [2]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult1.End(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.Sbmp.RedirectBindingElement.RedirectContainerChannelFactory1.RedirectContainerSessionChannel.EndRequest(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory1.RequestSessionChannel.RequestAsyncResult.<>c.<GetAsyncSteps>b__9_3(RequestAsyncResult thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.StepCallback(IAsyncResult result)

Exception rethrown at [3]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult1.End(IAsyncResult asyncResult) at Microsoft.ServiceBus.Messaging.Channels.ReconnectBindingElement.ReconnectChannelFactory1.RequestSessionChannel.EndRequest(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.SbmpTransactionalAsyncResult1.<>c.<GetAsyncSteps>b__18_3(TIteratorAsyncResult thisPtr, IAsyncResult a) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult1.StepCallback(IAsyncResult result)

Exception rethrown at [4]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.EndSendCommand(IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.<.ctor>b__4_1(IAsyncResult result, Boolean forceCleanUp) at Microsoft.ServiceBus.Messaging.BatchManager`1.PerformFlushAsyncResult.OnSingleOperationCompleted(IAsyncResult result)

Exception rethrown at [5]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.BatchManager1.BatchedObjectsAsyncResult.End(IAsyncResult result) at Microsoft.ServiceBus.Messaging.BatchManager1.EndBatchedOperation(IAsyncResult result) at Microsoft.ServiceBus.Messaging.BatchManagerAsyncResult`1.OnBatchedCallback(IAsyncResult result) at Microsoft.ServiceBus.Common.AsyncResult.AsyncCompletionWrapperCallback(IAsyncResult result)

Exception rethrown at [6]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.Sbmp.SbmpMessageSender.OnEndSend(IAsyncResult result) at Microsoft.ServiceBus.Messaging.MessageSender.RetrySenderAsyncResult.<>c.b__11_7(RetrySenderAsyncResult thisPtr, IAsyncResult r) at Microsoft.ServiceBus.Messaging.IteratorAsyncResult`1.StepCallback(IAsyncResult result)

Exception rethrown at [7]: at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.ServiceBus.Messaging.MessageSender.EndSendBatch(IAsyncResult result) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.Transport.AzureServiceBus.TaskWithRetryExtensions.d1.MoveNext() in Z:\BuildAgent\work\2f57832e2eee436e\src\Transport\Connectivity\TaskWithRetryExtensions.cs:line 18 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.Transport.AzureServiceBus.DefaultOutgoingBatchRouter.d14.MoveNext() in Z:\BuildAgent\work\2f57832e2eee436e\src\Transport\Sending\DefaultOutgoingBatchRouter.cs:line 0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at NServiceBus.Transport.AzureServiceBus.DefaultOutgoingBatchRouter.d__13.MoveNext() in Z:\BuildAgent\work\2f57832e2eee436e\src\Transport\Sending\DefaultOutgoingBatchRouter.cs:line 135

Packages.config

package id="NServiceBus" version="6.1.3" package id="NServiceBus.Autofac" version="6.0.1" package id="NServiceBus.Azure.Transports.WindowsAzureServiceBus" version="7.1.4" package id="NServiceBus.DataBus.AzureBlobStorage" version="1.0.1" package id="NServiceBus.Hosting.Azure" version="7.0.0" package id="NServiceBus.Persistence.AzureStorage" version="1.0.0"

Thanx in advance.

Greetings,

Joey Chömpff

SeanFeldman commented 7 years ago

@JSCProjects not sure if there's a connection with the databus...

But on one particular moment in time there were messages send to a queue without the databus header, so therefore it was impossible to receive it correct in the receiving handler.

How did this happen? Is that something that has some sort of condition you could reproduce? That would be an issue with the databus.

From the stack trace, it looks that the ASB transport has exhausted all the retries and was still receiving the TimeoutException. The TrackingId (edc36893-e81b-4da6-87f7-0662845a8d5c) should allow you to open a support case with the MSFT ASB team to get more details what happened with the request.

Is this something that you could reproduce in an isolated project and share? Thank you.

JSCProjects commented 7 years ago

@SeanFeldman I tried to reproduce it, but I can't figure out what the right situation is :-( But yesterday it happened again with 4 messages. I will open a request with MSFT ASB team, hopefully they can find something

JSCProjects commented 7 years ago

@SeanFeldman when is the databus header attached to the message? Is it possible that it's only attached when the property is containing data?

SeanFeldman commented 7 years ago

@JSCProjects databus header is attached when a property value is not null. In case the property value is null, the header will not be added upon sending. Upon receiving, if no header is found, no action will take place.

The exception you've seen has to do with the ASB failing to send messages on the client side and databus is just red-haring imo. Have you had any luck with the MSFT support?

Try to repro this as a stand-alone scenario to see if you can get the same issue again.

SeanFeldman commented 7 years ago

Closing this issue. Please reopen it if you have a repro that would indicate it's an Azure databus related failure. Thank you.

JSCProjects commented 7 years ago

NP. I’ve got enough evidence that it’s the problem of our legacy app

From: Sean Feldman [mailto:notifications@github.com] Sent: woensdag 19 april 2017 08:04 To: Particular/NServiceBus.DataBus.AzureBlobStorage NServiceBus.DataBus.AzureBlobStorage@noreply.github.com Cc: JSC Projects mail@jsc-projects.nl; Mention mention@noreply.github.com Subject: Re: [Particular/NServiceBus.DataBus.AzureBlobStorage] Incomplete message was send to the queue (#22)

Closing this issue. Please reopen it if you have a repro that would indicate it's an Azure databus related failure. Thank you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Particular/NServiceBus.DataBus.AzureBlobStorage/issues/22#issuecomment-295110195, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ASoNXdPayWbzjtnmcEb3_kj-wL5i4Qu3ks5rxaO8gaJpZM4MypL5.