Open ozzyaaron opened 2 years ago
Seeing some issues with redis 5.x.x gem and BACKGROUND=yes
too, scheduler exits silently with 0 code, but background process is not running, from what I gathered in strace -f
child process fails with something like:
"NoMethodError: undefined method `reconnect' for #<Redis::Client redis://localhost:6379/0>
I tried the latest version and it still silently fails in background mode (I wish logging wasn't so broken for resque-scheduler
). strace
shows some errors like:
rake aborted!
Redis::CannotConnectError: getaddrinfo_a: System error
...
Caused by:
RedisClient::CannotConnectError: getaddrinfo_a: System error
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/ruby_connection.rb:49:in `initialize'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `new'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `block in connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/middlewares.rb:8:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:652:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `block in connect'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `block in connect_with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `block in with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/tracer.rb:352:in `capture_segment_error'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `connect_with_tracing'
/.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `connect'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:646:in `raw_connection'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:613:in `ensure_connected'
/.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:212:in `call_v'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/client.rb:73:in `call_v'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:167:in `block in send_command'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `synchronize'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `send_command'
/.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/commands/connection.rb:21:in `ping'
/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
/.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
/.../shared/bundle/ruby/2.7.0/gems/resque-2.4.0/lib/resque/data_store.rb:83:in `reconnect'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:42:in `setup_backgrounding'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:17:in `setup'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/cli.rb:113:in `setup_env'
/.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/tasks.rb:17:in `block (2 levels) in <main>'
/.../shared/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => resque:scheduler
(See full trace by running task with --trace)
After upgrading from Redis gem v4.8.0 to v5.0.2 we got the following error on Heroku.
Our Redis URLs take the form
redis://h:pxxxxxx@an.amazon.server:12345
Resque.schedule = YAML.load_file(Rails.root.join("config", "resque_schedule.yml")) RedisClient::CommandError: ERR wrong number of arguments for 'auth' command
Reverting to Redis v4.8.0 corrects the issue.
I think Redis 5.0 changed the way you specify the server and port. I'm not near a computer though so that may be something you want to look into.
I tried the latest version and it still silently fails in background mode (I wish logging wasn't so broken for
resque-scheduler
).strace
shows some errors like:rake aborted! Redis::CannotConnectError: getaddrinfo_a: System error ... Caused by: RedisClient::CannotConnectError: getaddrinfo_a: System error /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/ruby_connection.rb:49:in `initialize' /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `new' /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:653:in `block in connect' /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client/middlewares.rb:8:in `connect' /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:652:in `connect' /.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `block in connect' /.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `block in connect_with_tracing' /.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `block in with_tracing' /.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/tracer.rb:352:in `capture_segment_error' /.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:45:in `with_tracing' /.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:30:in `connect_with_tracing' /.../shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.10.1/lib/new_relic/agent/instrumentation/redis/prepend.rb:20:in `connect' /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:646:in `raw_connection' /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:613:in `ensure_connected' /.../shared/bundle/ruby/2.7.0/gems/redis-client-0.9.0/lib/redis_client.rb:212:in `call_v' /.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/client.rb:73:in `call_v' /.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:167:in `block in send_command' /.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `synchronize' /.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis.rb:166:in `send_command' /.../shared/bundle/ruby/2.7.0/gems/redis-5.0.5/lib/redis/commands/connection.rb:21:in `ping' /.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace' /.../shared/bundle/ruby/2.7.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>' /.../shared/bundle/ruby/2.7.0/gems/resque-2.4.0/lib/resque/data_store.rb:83:in `reconnect' /.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:42:in `setup_backgrounding' /.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/env.rb:17:in `setup' /.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/cli.rb:113:in `setup_env' /.../shared/bundle/ruby/2.7.0/gems/resque-scheduler-4.7.0/lib/resque/scheduler/tasks.rb:17:in `block (2 levels) in <main>' /.../shared/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>' Tasks: TOP => resque:scheduler (See full trace by running task with --trace)
This one is a new one to me. I wish I knew what that system error was. Reverting to redis 4.8 fixes it?
This one is a new one to me. I wish I knew what that system error was. Reverting to redis 4.8 fixes it?
Yes.
FYI I fixed this by changing redis connection string from:
Resque.redis = 'localhost:6379'
to
Resque.redis = '127.0.0.1:6379'
Not sure what's the deal with getaddrinfo_a: System error
. It only happens when started by capistrano-redis
. I couldn't reproduce it manually.
After upgrading from Redis gem v4.8.0 to v5.0.2 we got the following error on Heroku.
Our Redis URLs take the form
redis://h:pxxxxxx@an.amazon.server:12345
Reverting to Redis v4.8.0 corrects the issue.