search

QueueClassic / queue_classic

Simple, efficient worker queue for Ruby & PostgreSQL.
MIT License
1.19k stars 153 forks source link

Queue Classic crashes regularly with "PG::ConnectionBad: connection is closed" #324

Open rmoriz opened 4 years ago

rmoriz commented 4 years ago

On production instance (rake qc:work) regularly crashes with PG::ConnectionBad: connection is closed exceptions.

This may be related to QueueClassic's https://github.com/QueueClassic/queue_classic/issues/317 and https://github.com/QueueClassic/queue_classic/pull/318 .

I'm not sure if this has any real world impact as long as systemd restards the worker, which it does.

see logs ``` Feb 13 12:33:24 app01 systemd[1]: app-queue.service: Scheduled restart job, restart counter is at 9. Feb 13 12:33:24 app01 systemd[1]: Stopped app-queue for app-backend. Feb 13 12:33:24 app01 systemd[1]: Started app-queue for app-backend. Feb 13 12:39:28 app01 app-queue[10513]: rake aborted! Feb 13 12:39:28 app01 app-queue[10513]: PG::ConnectionBad: connection is closed Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/conn_adapter.rb:36:in `exec' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/conn_adapter.rb:36:in `block in wait' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/conn_adapter.rb:31:in `synchronize' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/conn_adapter.rb:31:in `wait' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/worker.rb:95:in `lock_job' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/worker.rb:71:in `work' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/worker.rb:45:in `start' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/queue_classic-3.1.0/lib/queue_classic/tasks.rb:27:in `block (2 levels) in
' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/shared/bundle/2.6.2/ruby/2.6.0/gems/rake-13.0.0/exe/rake:27:in `' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/.asdf/installs/ruby/2.6.2/bin/bundle:23:in `load' Feb 13 12:39:28 app01 app-queue[10513]: /srv/app/.asdf/installs/ruby/2.6.2/bin/bundle:23:in `
' Feb 13 12:39:28 app01 app-queue[10513]: Tasks: TOP => qc:work Feb 13 12:39:28 app01 app-queue[10513]: (See full trace by running task with --trace) ```