search

mhenrixon / sidekiq-unique-jobs

Prevents duplicate Sidekiq jobs
MIT License
1.46k stars 277 forks source link

TypeError · Unsupported command argument type: Array #851

Open nickcoyne opened 1 month ago

nickcoyne commented 1 month ago

Describe the bug

I have a strange startup bug with this error: Unsupported command argument type: Array

vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/command_builder.rb:37:in `block in generate': Unsupported command argument type: Array

            raise TypeError, "Unsupported command argument type: #{element.class}"
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TypeError)
    from vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/command_builder.rb:28:in `map!'
    from vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/command_builder.rb:28:in `generate'
    from vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client.rb:276:in `call'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/redis_client_adapter.rb:21:in `evalsha'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/scripts.rb:115:in `execute'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/client.rb:45:in `block in execute'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/timing.rb:20:in `timed'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/client.rb:44:in `execute'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/dsl.rb:26:in `execute'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/caller.rb:59:in `do_call'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/caller.rb:48:in `block in call_script'
    from vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/decorator.rb:46:in `block in with'
    from vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client.rb:241:in `with'
    from vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/decorator.rb:46:in `with'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/connection.rb:16:in `block in redis'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/config.rb:167:in `block in redis'
    from vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
    from vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
    from vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
    from vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
    from vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/config.rb:164:in `redis'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq.rb:74:in `redis'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/connection.rb:15:in `redis'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/script/caller.rb:47:in `call_script'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/update_version.rb:18:in `call'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/server.rb:37:in `start'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-unique-jobs-8.0.10/lib/sidekiq_unique_jobs/server.rb:22:in `block in configure'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/component.rb:60:in `block in fire_event'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/component.rb:59:in `each'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/component.rb:59:in `fire_event'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/cli.rb:108:in `run'
    from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/bin/sidekiq:31:in `<top (required)>'
    from vendor/bundle/ruby/3.2.0/bin/sidekiq:25:in `load'
    from vendor/bundle/ruby/3.2.0/bin/sidekiq:25:in `<top (required)>'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/cli/exec.rb:58:in `load'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/cli/exec.rb:58:in `kernel_load'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/cli/exec.rb:23:in `run'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/cli.rb:451:in `exec'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/cli.rb:34:in `dispatch'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/cli.rb:28:in `start'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/exe/bundle:28:in `block in <top (required)>'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    from vendor/bundle/ruby/3.2.0/gems/bundler-2.5.6/exe/bundle:20:in `<top (required)>'
    from bin/bundle:104:in `load'
    from bin/bundle:104:in `<main>'

Expected behavior

Sidekiq starts without issue

Current behavior

Sidekiq startup fails.

Additional context

I'm upgrading my Rails 6.1.7.8 application to use Sidekiq 7 and friends (previously sidekiq 6.5.12, and sidekiq-unique-jobs 7.1.33, now sidekiq 7.3.0 and sidekiq-unique-jobs 8.0.10).

On sidekiq worker boot it fails with Unsupported command argument type: Array

It is raised from vendor/bundle/ruby/3.2.0/gems/redis-client-0.22.2/lib/redis_client/command_builder.rb:37

Locally, if I delay sidekiq startup to occur a few seconds after redis startup, then the error does not occur. This also works by delaying the in the sidekiq config like so:

  Thread.new do
    sleep 2

    SidekiqUniqueJobs::Server.configure(config)
  end

However that doesn't seem to help with my Heroku restarts where I still get the error. Manually restarting the worker on Heroku usually works, although sometimes with this error:

NoMethodError · undefined method `lines' for {"server"=>"redis", "version"=>"6.2.8", "proto"=>3, "id"=>3684324, "mode"=>"standalone", "role"=>"master", "modules"=>[]}:Hash conn.call("INFO") { \|i\| i.lines(chomp: true).map { \|l\| l.split(":", 2) }.select { \|l\| l.size == 2 }.to_h }

which is coming from vendor/bundle/ruby/3.2.0/gems/sidekiq-7.3.0/lib/sidekiq/config.rb:148

If I omit the SidekiqUniqueJobs::Server.configure(config) completely, then there are no startup issues.

Running ruby 3.2.4. Redis versions are 7.2.5 locally and 6.2.8 on Heroku.

nickcoyne commented 1 month ago

This is my full sidekiq config:

require 'sidekiq'
require 'sidekiq-unique-jobs'

redis_options = {
  network_timeout: 5
}

Sidekiq.configure_client do |config|
  config.redis = redis_options

  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end
end

Sidekiq.configure_server do |config|
  config.redis = redis_options

  config.client_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Client
  end

  config.server_middleware do |chain|
    chain.add SidekiqUniqueJobs::Middleware::Server
  end

  # # This is a workaround for redis connection issues on boot.
  Thread.new do
    sleep 2

    SidekiqUniqueJobs::Server.configure(config)
  end
end