search

pardahlman / RawRabbit

A modern .NET framework for communication over RabbitMq
MIT License
747 stars 144 forks source link

cannot obtain exclusive access to locked queue #192

Closed PeterKottas closed 7 years ago

PeterKottas commented 7 years ago

Hi mate hope you are well. I've been using the lib for a while all going well but now encountered something that might require an expert :) Here's the deal. I have a service and somewhere along the way I have client.RespondAsync<GetHistoricalRacecardRequestDTO, GetHistoricalRacecardResponseDTO>(GetRacecard); where private async Task<GetHistoricalRacecardResponseDTO> GetRacecard(GetHistoricalRacecardRequestDTO request) All works fine with one service running but when scaling horizontally I get The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=405, text="RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'gethistoricalracecardrequestdto' in vhost '/'", classId=50, methodId=10, cause=

The problem makes sense to me ... there is already one listener and it locked the queue. But I am trying to figure out what would the best practice be to enable having multiple clients listening and reacting to messages being sent. Maybe config? Or pass something to the client. Not sure.

PeterKottas commented 7 years ago

I actually have custom wrapper around the lib that is externally unaware of the methods and objects so would be great if it was config. But maybe not possible.

PeterKottas commented 7 years ago

And as a side note, this exception caught and re thrown actually keeps the app hanging. That's kind of worrying as well. I've "fixed" it by adding Environment.Exit(0); Although that might be something to look into as well. It might be this although that's obviously just a guess.

pardahlman commented 7 years ago

Hello @PeterKottas - all well here! Scaling horizontally should work out of the box, we're running a similar setup ourselves without any problems. Are you running the 2.0 beta or 1.10.3?

Based on the error message, it sounds like the queue is declared as Exclusive. Are you using the fluent configuration builder to mark the queue as exclusive, or have you registered a custom RawRabbitConfiguration (either through code or config) with Exclusive set to true?

As for your side note, in order to understand why this happens, I would need to know more about your setup. The exception should propagates to the caller. Perhaps you could create a sample project where you get this behavior and share with me?

PeterKottas commented 7 years ago

Ok so using 1.10.3 over here. And you are 100% correct, I've got wrong setting in json. "Queue": { "AutoDelete": true, "Durable": true, "Exclusive": true } Changing it to false helped right away and solved the problem. Thanks. I'll see what I can do about the code sharing. It's the company code so it's not totally straight forward. The exception propagates just fine but it stays hanging on something. The setup is quite complex but it seems that removing rawrabbit and throwing an "on demand" exception works just fine.

pardahlman commented 7 years ago

OK, glad to help!

Is the exception unhandled by the application? If so, I'm guessing that the bus client still holds reference to undisposed resources (the connection to the broker), which is why it's hanging. The solution is to dispose the bus client upon application shutdown. How you do this depends on how you host your application and general lifetime mgmt.

(Looking at your configuration, I see that the queue has AutoDelete set to true. This is OT, but RawRabbit defaults to false for this value, since it in many scenarios makes sense to keep the queue even though there is no consumers on it. If AutoDelete is set to true and all consumer application crashes, the queue is deleted and new messages wont be routed to that queue (as it is deleted). But then again, this might be intended behavior in your application)

Don't stress about code sharing at this point! Cheers!

PeterKottas commented 7 years ago

Thanks again, yep the autodelete is fine that's intentional just reconfigured Exclusive. I'm using autofac and fixed the "OnRelease" to call ShutdownAsync also configuring the GracfulShutdown or what's the name of it to force it to go faster. I got me at first that it took so long and now all is ticking :) I've noticed that there's IDisposable implemented on the client. Autofac calls it automatically link but for some reason, it doesn't shut down the client. Or it might be something else. The app is becoming quite complex so I might be wrong and something else might be keeping it alive. If you're sure that Dispose finishes it off properly, I think we can just leave this :)