search

rebus-org / Rebus.AzureServiceBus

:bus: Azure Service Bus transport for Rebus
https://mookid.dk/category/rebus
Other
33 stars 20 forks source link

Error registering event subscriptions when running multiple instances #19

Closed MortenChristiansen closed 5 years ago

MortenChristiansen commented 5 years ago

We get a Microsoft.Azure.ServiceBus.ServiceBusException with the message "Another update request is in progress for the entity X" when trying to register subscriptions for the same events in multiple websites. We are running multiple instances of our web service so they run the same initialization code. Is this a problem that Rebus should deal with or must we coordinate between our instances to figure out which one registers the handlers... or something else entirely?


at Buma.Host.BoundedContext.AppExtensions.InitializeHost[TCommand](IApplicationBuilder app, String serviceName, Boolean isBoundedContext, Measurement measurement)
   at Buma.Host.BoundedContext.BumaBoundedContextStartup`1.Configure(IApplicationBuilder app, IHostingEnvironment env, IApplicationLifetime applicationLifetime)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
---> (Inner Exception #0) Microsoft.Azure.ServiceBus.ServiceBusException: Another update request is in progress for the entity studicabus-dev:Topic:admission_events_schools_schoolimportedevent__admission_events|studica_denormalizer_admission_shared. TrackingId:b8d9a853-9dd6-47f8-872b-010d652d44bb_G1_B32, SystemTracker:studicabus-dev:Topic:admission_events_schools_schoolimportedevent__admission_events|studica_denormalizer_admission_shared, Timestamp:11/20/2018 12:14:29 PM
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.PutEntity(String path, String requestBody, Boolean isUpdate, String forwardTo, String fwdDeadLetterTo, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.UpdateSubscriptionAsync(SubscriptionDescription subscriptionDescription, CancellationToken cancellationToken)
   at Rebus.AzureServiceBus.AzureServiceBusTransport.RegisterSubscriber(String topic, String subscriberAddress)
   at Rebus.Bus.RebusBus.InnerSubscribe(String topic)
   at Buma.Common.Rebus.BusSetup.SubscribeWithRetry(IBus bus, Type eventType)<---

---> (Inner Exception #1) Microsoft.Azure.ServiceBus.ServiceBusException: Another update request is in progress for the entity studicabus-dev:Topic:admission_events_applications_eudapplicationimportedevent__admission_events|studica_denormalizer_admission_shared. TrackingId:4b0b2118-2d21-4c4b-9c58-3db7a5e515f2_G41_B45, SystemTracker:studicabus-dev:Topic:admission_events_applications_eudapplicationimportedevent__admission_events|studica_denormalizer_admission_shared, Timestamp:11/20/2018 12:14:30 PM
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.PutEntity(String path, String requestBody, Boolean isUpdate, String forwardTo, String fwdDeadLetterTo, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.UpdateSubscriptionAsync(SubscriptionDescription subscriptionDescription, CancellationToken cancellationToken)
   at Rebus.AzureServiceBus.AzureServiceBusTransport.RegisterSubscriber(String topic, String subscriberAddress)
   at Rebus.Bus.RebusBus.InnerSubscribe(String topic)
   at Buma.Common.Rebus.BusSetup.SubscribeWithRetry(IBus bus, Type eventType)<---

Unhandled Exception: System.AggregateException: One or more errors occurred. (Another update request is in progress for the entity studicabus-dev:Topic:admission_events_schools_schoolimportedevent__admission_events|studica_denormalizer_admission_shared. TrackingId:b8d9a853-9dd6-47f8-872b-010d652d44bb_G1_B32, SystemTracker:studicabus-dev:Topic:admission_events_schools_schoolimportedevent__admission_events|studica_denormalizer_admission_shared, Timestamp:11/20/2018 12:14:29 PM) (Another update request is in progress for the entity studicabus-dev:Topic:admission_events_applications_eudapplicationimportedevent__admission_events|studica_denormalizer_admission_shared. TrackingId:4b0b2118-2d21-4c4b-9c58-3db7a5e515f2_G41_B45, SystemTracker:studicabus-dev:Topic:admission_events_applications_eudapplicationimportedevent__admission_events|studica_denormalizer_admission_shared, Timestamp:11/20/2018 12:14:30 PM) ---> Microsoft.Azure.ServiceBus.ServiceBusException: Another update request is in progress for the entity studicabus-dev:Topic:admission_events_schools_schoolimportedevent__admission_events|studica_denormalizer_admission_shared. TrackingId:b8d9a853-9dd6-47f8-872b-010d652d44bb_G1_B32, SystemTracker:studicabus-dev:Topic:admission_events_schools_schoolimportedevent__admission_events|studica_denormalizer_admission_shared, Timestamp:11/20/2018 12:14:29 PM
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.PutEntity(String path, String requestBody, Boolean isUpdate, String forwardTo, String fwdDeadLetterTo, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.Management.ManagementClient.UpdateSubscriptionAsync(SubscriptionDescription subscriptionDescription, CancellationToken cancellationToken)
   at Rebus.AzureServiceBus.AzureServiceBusTransport.RegisterSubscriber(String topic, String subscriberAddress)
   at Rebus.Bus.RebusBus.InnerSubscribe(String topic)
   at Buma.Common.Rebus.BusSetup.SubscribeWithRetry(IBus bus, Type eventType)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAllCore(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Buma.Common.Rebus.BusSetup.SubscribeToHandledEvents(IBus bus, List`1 eventTypes)
   at Buma.Common.Rebus.BusSetup.InitBasicBus(ServiceType serviceType, IContainer container, IContainerAdapter activator, Measurement measurement)
   at Buma.Common.Rebus.BusSetup.InitDenormalizationBus(Measurement measurement)
   at Buma.Host.BoundedContext.AppExtensions.InitializeHost[TCommand](IApplicationBuilder app, String serviceName, Boolean isBoundedContext, Measurement measurement)
   at Buma.Host.BoundedContext.BumaBoundedContextStartup`1.Configure(IApplicationBuilder app, IHostingEnvironment env, IApplicationLifetime applicationLifetime)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.Configure(IApplicationBuilder app)
   at Microsoft.AspNetCore.HostFilteringStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder app)
   at Microsoft.AspNetCore.Hosting.Internal.AutoRequestServicesStartupFilter.<>c__DisplayClass0_0.<Configure>b__0(IApplicationBuilder builder)
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
   at Microsoft.AspNetCore.Hosting.Internal.WebHost.StartAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token, String shutdownMessage)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(IWebHost host, CancellationToken token)
   at Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host)
   at Admission.Host.Program.Main(String[] args) in /app/src/Admission/Admission.Host/Program.cs:line 11
Aborted (core dumped)
mookid8000 commented 5 years ago

Wow, I didn't know that could happen.

While I would definitely have wished that the Azure Service Bus driver handled this internally somehow, I would say that Rebus (i.e. the Azure Service Bus transport) should handle that and automatically retry the operation.

Stay tuned 😄

mookid8000 commented 5 years ago

I've pushed Rebus.AzureServiceBus 7.0.0-a02 that should fix this 😄

I couldn't reproduce the concurrency issues demonstrated here though, so I assumed that a ServiceBusException would be thrown with IsTransient == true.

If this is not the case, then it will not work 😐 in that case, please report back here, and then I'll make the necessary adjustments.

Thanks for reporting this issue! 👍

MortenChristiansen commented 5 years ago

Right now we can't reproduce the error, so I assume the problem is solved. I'll let you know if we trigger it again.