search

wagslane / go-rabbitmq

A wrapper of streadway/amqp that provides reconnection logic and sane defaults
https://blog.boot.dev/golang/connecting-to-rabbitmq-in-golang-easy/
MIT License
746 stars 125 forks source link

only reconnecting some consumers #137

Open MaxBreida opened 11 months ago

MaxBreida commented 11 months ago

Hey, we encounter some problems with the reconnection of consumers in our services. It seems that only some of them are being recovered, sometimes even none at all. 

Initializing environment variables...
Initializing queues...
2023-07-25T13:40:02+02:00 ??? 2023/07/25 13:40:02 gorabbit INFO: Processing messages on 1 goroutines
2023-07-25T13:40:03+02:00 ??? 2023/07/25 13:40:03 gorabbit INFO: Processing messages on 1 goroutines
2023-07-25T13:40:03+02:00 ??? 2023/07/25 13:40:03 gorabbit INFO: Processing messages on 1 goroutines
2023-07-25T13:40:03+02:00 ??? 2023/07/25 13:40:03 gorabbit INFO: Processing messages on 1 goroutines
start consuming events
Started to listen for sigkill and sigterm
2023-07-25T13:46:18+02:00 ??? 2023/07/25 13:46:18 gorabbit INFO: rabbit consumer goroutine closed
2023-07-25T13:46:18+02:00 ??? 2023/07/25 13:46:18 gorabbit ERROR: attempting to reconnect to amqp server after connection close with error: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T13:46:18+02:00 ??? 2023/07/25 13:46:18 gorabbit INFO: waiting 5s seconds to attempt to reconnect to amqp server
2023-07-25T13:46:18+02:00 ??? 2023/07/25 13:46:18 gorabbit ERROR: attempting to reconnect to amqp server after close with error: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T13:46:18+02:00 ??? 2023/07/25 13:46:18 gorabbit ERROR: attempting to reconnect to amqp server after close with error: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T13:46:18+02:00 ??? 2023/07/25 13:46:18 gorabbit INFO: waiting 5s seconds to attempt to reconnect to amqp server
2023-07-25T13:46:18+02:00 ??? 2023/07/25 13:46:18 gorabbit INFO: waiting 5s seconds to attempt to reconnect to amqp server
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit WARN: error closing connection while reconnecting: Exception (504) Reason: "channel/connection is not open"
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit WARN: successfully reconnected to amqp server
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit INFO: successful connection recovery from: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit WARN: error closing channel while reconnecting: Exception (504) Reason: "channel/connection is not open"
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit WARN: successfully reconnected to amqp server
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit INFO: successful publisher recovery from: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit WARN: error closing channel while reconnecting: Exception (504) Reason: "channel/connection is not open"
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit WARN: successfully reconnected to amqp server
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit INFO: successful consumer recovery from: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T13:46:23+02:00 ??? 2023/07/25 13:46:23 gorabbit INFO: Processing messages on 1 goroutines
recreating channel cause of error: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 connection closed, reason: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 Exception (504) Reason: "channel/connection is not open"
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 channel closed, reason: %vException (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 gorabbit INFO: rabbit consumer goroutine closed
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 gorabbit ERROR: attempting to reconnect to amqp server after connection close with error: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 gorabbit INFO: waiting 5s seconds to attempt to reconnect to amqp server
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 gorabbit ERROR: attempting to reconnect to amqp server after close with error: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 gorabbit ERROR: attempting to reconnect to amqp server after close with error: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 gorabbit INFO: waiting 5s seconds to attempt to reconnect to amqp server
2023-07-25T14:10:16+02:00 ??? 2023/07/25 14:10:16 gorabbit INFO: waiting 5s seconds to attempt to reconnect to amqp server
2023-07-25T14:10:19+02:00 ??? 2023/07/25 14:10:19 channel recreate failed, err: Exception (504) Reason: "channel/connection is not open"
2023-07-25T14:10:19+02:00 ??? 2023/07/25 14:10:19 reconnect success
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit WARN: error closing connection while reconnecting: Exception (504) Reason: "channel/connection is not open"
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit WARN: successfully reconnected to amqp server
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit INFO: successful connection recovery from: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit WARN: error closing channel while reconnecting: Exception (504) Reason: "channel/connection is not open"
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit WARN: successfully reconnected to amqp server
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit INFO: successful consumer recovery from: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit WARN: error closing channel while reconnecting: Exception (504) Reason: "channel/connection is not open"
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit WARN: successfully reconnected to amqp server
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit INFO: successful publisher recovery from: Exception (320) Reason: "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'"
2023-07-25T14:10:21+02:00 ??? 2023/07/25 14:10:21 gorabbit INFO: Processing messages on 1 goroutines
2023-07-25T14:10:22+02:00 ??? 2023/07/25 14:10:22 channel recreate success
2023-07-25T14:24:37+02:00 INF successfully handled event: xyz service=zyx <---- from one consumer
--- no further messages after that ---

As you can see the service started with 4 consumers but only one seems to have reconnected. This seems to be a similar or even the same issue as #99. I've also checked #87 but we're already using only one consumer per queue.

Should the consumers have each their own connection (#79)? Maybe I've misunderstood something but I thought with the connectionManager and channelManager there shouln't be a need for this. Using v0.12.4

wagslane commented 4 months ago

The related issues are closed, are you still seeing this problem with the latest version?