search

wagslane / go-rabbitmq

A wrapper of streadway/amqp that provides reconnection logic and sane defaults
https://blog.boot.dev/golang/connecting-to-rabbitmq-in-golang-easy/
MIT License
768 stars 125 forks source link

Reasonable default for QOS prefetch? #113

Closed ae-govau closed 1 year ago

ae-govau commented 1 year ago

I started using this library recently for a Go app to replace a Java app. I thought I'd start with as minimal configuration as possible, and then tune as I went.

The examples are good, and my code functionally worked as expected, but when left overnight running in an environment with a larger message load, I found the next day so it had slowed right down, nearly to a halt, and saw various out of memory related errors and restarts.

Eventually I noticed the Rabbit MGMT UI showing that all 400,000 messages in the queue were "Unacked" (rather than "Ready") and I realised that the default prefetch of 0, which I'd assumed meant "prefetch 0 messages" actually meant "prefetch UNLIMITED messages". And as such my app prefetched all of them, and then bad things happened (due to low-memory).

Noting the objective of this library to provide sane defaults - I wonder if unlimited prefetch is a sane default?

(more generally I'm interested to know how well prefetch interacts with low-memory situations, e.g. I'm curious if there's anything to reduce prefetching, and/or reduce concurrency, if approaching GOMEMLIMIT or similar...)

ae-govau commented 1 year ago

Thanks @wagslane ! (and thanks for the great library)

Mikaelemmmm commented 9 months ago

I'm having the same problem as you on my end, what is the approximate value of this value configured according to this amount of messages you have? Thank you for your help.