Closed haydenm315 closed 6 months ago
Hey @haydenm315 π Thanks for the report π I forgot to ask you about the RR version. Could you please try the latest one (2023.3.9)?
Just tried with 2023.3.10 and there's messages which indicate an attempt to recover, however I still don't see the queue listed when doing ./rr workers, after bouncing the rabbitmq service to simulate an upgrade or service interruption. Looks like there's an attempt to re-establish the connection now.
Upon making rabbitmq unavailable, roadrunner logs the following.
roadrunner-1 | {"level":"DEBUG","ts":"2024-02-07T18:14:21+0000","logger":"amqp ","msg":"delivery channel was closed, leaving the rabbit listener"}
roadrunner-1 | {"level":"DEBUG","ts":"2024-02-07T18:14:21+0000","logger":"amqp ","msg":"number of listeners","listeners":0}
roadrunner-1 | {"level":"DEBUG","ts":"2024-02-07T18:14:21+0000","logger":"amqp ","msg":"exited from redialer"}
roadrunner-1 | {"level":"ERROR","ts":"2024-02-07T18:14:21+0000","logger":"amqp ","msg":"amqp connection closed","error":"Exception (504) Reason: \"channel/connection is not open\""}
roadrunner-1 | {"level":"ERROR","ts":"2024-02-07T18:14:21+0000","logger":"amqp ","msg":"pipeline connection was closed, redialing","error":"Exception (320) Reason: \"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'\"","pipeline":"snapshot","driver":"amqp","start":"2024-02-07T18:14:21+0000"}
After making rabbitmq available again, roadrunner logs the following
roadrunner-1 | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"rabbitmq dial was succeed. trying to redeclare queues and subscribers"}
roadrunner-1 | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"queues and subscribers was redeclared successfully"}
roadrunner-1 | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"connection was successfully restored","pipeline":"snapshot","driver":"amqp","start":"2024-02-07T18:14:21+0000","elapsed":"27.519989644s"}
roadrunner-1 | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"redialer restarted"}
roadrunner-1 | {"level":"DEBUG","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"exited from redialer"}
roadrunner-1 | {"level":"ERROR","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"pipeline connection was closed, redialing","error":"Exception (320) Reason: \"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'\"","pipeline":"snapshot","driver":"amqp","start":"2024-02-07T18:14:49+0000"}
roadrunner-1 | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"rabbitmq dial was succeed. trying to redeclare queues and subscribers"}
roadrunner-1 | {"level":"INFO","ts":"2024-02-07T18:14:49+0000","logger":"amqp ","msg":"rabbitmq dial was succeed. trying to redeclare queues and subscribers"}
Yeah, so, that means that RR successfully redialed.
@haydenm315 Is that what you wanted to see, am I right?
Seems closer than the older release we were using. Still requires a restart of roadrunner to get it to accept jobs and have a ready status. Unless I restart roadrunner after rabbitmq comes back, I don't see jobs like below in the rr workers output.
Jobs of [jobs]:
+--------+----------+--------+---------+--------+---------+----------+
| STATUS | PIPELINE | DRIVER | QUEUE | ACTIVE | DELAYED | RESERVED |
+--------+----------+--------+---------+--------+---------+----------+
| READY | snapshot | amqp | default | 0 | 0 | 0 |
+--------+----------+--------+---------+--------+---------+----------+
Maybe something's not quite right with "trying to redeclare queues and subscribers".
I'm not sure, because I have a test especially for the case when rabbitmq is down. Have you tried to push the job after redial?
@haydenm315 In your configuration I also see, that you're consuming pipeline with the name widgets
, but you don't have such a pipeline in your pipelines configuration.
I renamed the queue in the previous example. It's not a queue naming issue. Jobs are getting pushed because when I restart the roadrunner server the number of Execs goes up.
But, it's not a queue, it's a pipeline. Could you please share the latest configuration you have?
Here the .rr.yaml file These are the env var values in the cfg file
AMQP_REDIAL_TIMEOUT=99999999 ROADRUNNER_NUM_POLLERS=200 ROADRUNNER_NUM_WORKERS=50 ROADRUNNER_PREFETCH=200
Thanks π Few notes regarding your configuration:
pollers
should not be significantly bigger than the number of workers. Poller is a thread inside RR which tries to obtain a job from the internal priority queue. More pollers -> more memory will be used for the thread stack, etc. Try 60 pollers
and compare with the current value (200).reload
plugin anymore since 2023.1.0 version.Thanks for the recommendations
Hey @haydenm315 π
I double-checked that behavior. Let me summarize what I did:
Keep in mind, that RR uses an exponential backoff mechanism to redial. It won't reconnect immediately, but after some exponential step. For example: first redial will happen after 1sec after loosing a connection, next - 2s -> 5s -> 15s -> 25s ...
I also noticed, that your amqp
configuration contains some options from the other drivers, e.g.: reserve_timeout
is from the beanstalk
driver.
Closing as answered. You are welcome to comment here if you still have questions π
No duplicates π₯².
What happened?
I'm using roadrunner with rabbitmq amqp and jobs to run a pipeline. If there's an interruption in the rabbitmq broker such as restarting it, roadrunner fails to re-establish the connecftion to the rabbitmq broker queue.
I see one redialing attempt when the broker is restarted, however it's never reconnected.
Version (rr --version)
rr version 2023.1.5 (build time: 2023-06-08T14:45:05+0000, go1.20.4), OS: linux, arch: amd64
How to reproduce the issue?
To reproduce, start up a rabbitmq broker and then the roadrunner.
Verify the roadrunner is connected to the broker and is ready for work ./rr workers
Output will include the following at the bottom.
restart your rabbitmq broker roadrunner will output some errors as pasted below in the log output section.. After these errors are output, there won't be any activity.
Looking at the workers with ./rr workers won't have any job status, even though the rabbitmq broker is available. Only after restarting the roadrunner process will the amqp connect restore.
Relevant log output