resque / resque-scheduler

A light-weight job scheduling system built on top of Resque
MIT License
1.74k stars 481 forks source link

Use with Redis 5.0 gem causes error loading schedule #750

Open ozzyaaron opened 2 years ago

ozzyaaron commented 2 years ago

After upgrading from Redis gem v4.8.0 to v5.0.2 we got the following error on Heroku.

Our Redis URLs take the form redis://h:pxxxxxx@an.amazon.server:12345

Resque.schedule = YAML.load_file(Rails.root.join("config", "resque_schedule.yml"))

RedisClient::CommandError: ERR wrong number of arguments for 'auth' command

Reverting to Redis v4.8.0 corrects the issue.

xeron commented 2 years ago

Seeing some issues with redis 5.x.x gem and BACKGROUND=yes too, scheduler exits silently with 0 code, but background process is not running, from what I gathered in strace -f child process fails with something like:

"NoMethodError: undefined method `reconnect' for #<Redis::Client redis://localhost:6379/0>
xeron commented 2 years ago

I tried the latest version and it still silently fails in background mode (I wish logging wasn't so broken for resque-scheduler). strace shows some errors like:

rake aborted!

Redis::CannotConnectError: getaddrinfo_a: System error

...

Caused by:

RedisClient::CannotConnectError: getaddrinfo_a: System error

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/ruby_connection.rb:49:in `initialize'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `new'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `block in connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/middlewares.rb:8:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:652:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `block in connect'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `block in connect_with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `block in with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/tracer.rb:352:in `capture_segment_error'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `connect_with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:646:in `raw_connection'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:613:in `ensure_connected'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:212:in `call_v'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/client.rb:73:in `call_v'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:167:in `block in send_command'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `synchronize'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `send_command'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/commands/connection.rb:21:in `ping'
/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
/.../shared/bundle/ruby/2.7.0/gems/resque-2.4.0/lib/resque/data_store.rb:83:in `reconnect'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:42:in `setup_backgrounding'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:17:in `setup'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/cli.rb:113:in `setup_env'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/tasks.rb:17:in `block (2 levels) in <main>'
/.../shared/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'

Tasks: TOP => resque:scheduler

(See full trace by running task with --trace)
PatrickTulskie commented 2 years ago

After upgrading from Redis gem v4.8.0 to v5.0.2 we got the following error on Heroku.

Our Redis URLs take the form redis://h:pxxxxxx@an.amazon.server:12345


Resque.schedule = YAML.load_file(Rails.root.join("config", "resque_schedule.yml"))

RedisClient::CommandError: ERR wrong number of arguments for 'auth' command

Reverting to Redis v4.8.0 corrects the issue.

I think Redis 5.0 changed the way you specify the server and port. I'm not near a computer though so that may be something you want to look into.

PatrickTulskie commented 2 years ago

I tried the latest version and it still silently fails in background mode (I wish logging wasn't so broken for resque-scheduler). strace shows some errors like:


rake aborted!

Redis::CannotConnectError: getaddrinfo_a: System error

...

Caused by:

RedisClient::CannotConnectError: getaddrinfo_a: System error

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/ruby_connection.rb:49:in `initialize'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `new'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `block in connect'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/middlewares.rb:8:in `connect'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:652:in `connect'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `block in connect'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `block in connect_with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `block in with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/tracer.rb:352:in `capture_segment_error'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `connect_with_tracing'

/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `connect'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:646:in `raw_connection'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:613:in `ensure_connected'

/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:212:in `call_v'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/client.rb:73:in `call_v'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:167:in `block in send_command'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `synchronize'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `send_command'

/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/commands/connection.rb:21:in `ping'

/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'

/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'

/.../shared/bundle/ruby/2.7.0/gems/resque-2.4.0/lib/resque/data_store.rb:83:in `reconnect'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:42:in `setup_backgrounding'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:17:in `setup'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/cli.rb:113:in `setup_env'

/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/tasks.rb:17:in `block (2 levels) in <main>'

/.../shared/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'

Tasks: TOP => resque:scheduler

(See full trace by running task with --trace)

This one is a new one to me. I wish I knew what that system error was. Reverting to redis 4.8 fixes it?

xeron commented 2 years ago

This one is a new one to me. I wish I knew what that system error was. Reverting to redis 4.8 fixes it?

Yes.

xeron commented 1 year ago

FYI I fixed this by changing redis connection string from:

Resque.redis = 'localhost:6379'

to

Resque.redis = '127.0.0.1:6379'

Not sure what's the deal with getaddrinfo_a: System error. It only happens when started by capistrano-redis. I couldn't reproduce it manually.