deanpcmad / sidekiq-limit_fetch

A Sidekiq plugin to support advanced queue control (limiting, pausing, blocking, querying)
MIT License
947 stars 140 forks source link

Broken with Sidekiq 7? #142

Closed xdmx closed 1 year ago

xdmx commented 2 years ago

I've just tried to upgrade to Sidekiq 7 and I'm getting an error when Sidekiq starts:

undefined method `[]=' for #<Sidekiq::Capsule:0x00007f66a1219760 @name="default", @config=#<Sidekiq::Config:0x00007f66a1225628 @options={:labels=>#<Set: {}>, :require=>".", :environment=>nil, :concurrency=>5, :timeout=>25, :poll_interval_average=>nil, :average_scheduled_poll_interval=>5, :on_complex_arguments=>:raise, :error_handlers=>[#<Proc:0x00007f66a1ca0df0 /var/deploy/myapp/web_head/shared/bundle/ruby/3.1.0/gems/sidekiq-7.0.0/lib/sidekiq/config.rb:36 (lambda)>, #<Sentry::Sidekiq::ErrorHandler:0x00007f669ce331b8>], :death_handlers=>[], :lifecycle_events=>{:startup=>[], :quiet=>[], :shutdown=>[], :heartbeat=>[], :beat=>[#<Proc:0x00007f66a1224ed0 /var/deploy/myapp/web_head/shared/bundle/ruby/3.1.0/gems/sidekiq-7.0.0/lib/sidekiq/metrics/tracking.rb:131>]}, :dead_max_jobs=>10000, :dead_timeout_in_seconds=>15552000, :reloader=>#<Sidekiq::Rails::Reloader @app=Myapp::Application>, :dynamic=>true, :queues=>["mailers", "critical", "default", "low"], :config_file=>"config/sidekiq.yml", :limits=>{}, :tag=>"web_head", :identity=>"pigeon:510451:451380f36ea4"}, @directory={}, @redis_config={}, @capsules={"default"=>#<Sidekiq::Capsule:0x00007f66a1219760 ...>}, @server_chain=#<Sidekiq::Middleware::Chain:0x00007f66a12251f0 @config=#<Sidekiq::Config:0x00007f66a1225628 ...>, @entries=[#<Sidekiq::Middleware::Entry:0x00007f66a1224fc0 @config=#<Sidekiq::Config:0x00007f66a1225628 ...>, @klass=Sidekiq::Metrics::Middleware, @args=[#<Sidekiq::Metrics::ExecutionTracker:0x00007f66a1225470 @config=#<Sidekiq::Config:0x00007f66a1225628 ...>, @jobs={}, @totals={}, @grams={}, @lock=#<Thread::Mutex:0x00007f66a1225290>>]>, #<Sidekiq::Middleware::Entry:0x00007f669ce32f60 @config=#<Sidekiq::Config:0x00007f66a1225628 ...>, @klass=Sentry::Sidekiq::SentryContextServerMiddleware, @args=[]>]>, @client_chain=#<Sidekiq::Middleware::Chain:0x00007f669ce32e70 @config=#<Sidekiq::Config:0x00007f66a1225628 ...>, @entries=[#<Sidekiq::Middleware::Entry:0x00007f669ce32d58 @config=#<Sidekiq::Config:0x00007f66a1225628 ...>, @klass=Sentry::Sidekiq::SentryContextClientMiddleware, @args=[]>]>, @logger=#<Sidekiq::Logger:0x00007f6699366b70 @level=1, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f6699366a08 @datetime_format=nil>, @formatter=#<Sidekiq::Logger::Formatters::Pretty:0x00007f6699366760 @datetime_format=nil>, @logdev=#<Logger::LogDevice:0x00007f6699366918 @shift_period_suffix=nil, @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<IO:<STDOUT>>, @binmode=false, @mon_data=#<Monitor:0x00007f66993668c8>, @mon_data_owner_object_id=21040>, @skip_instrumenting=true>, @redis=#<ConnectionPool:0x00007f66981707e0 @size=5, @timeout=1, @available=#<ConnectionPool::TimedStack:0x00007f6698170308 @create_block=#<Proc:0x00007f6698170718 /var/deploy/myapp/web_head/shared/bundle/ruby/3.1.0/gems/sidekiq-7.0.0/lib/sidekiq/redis_connection.rb:22>, @created=1, @que=[#<Sidekiq::RedisClientAdapter::CompatClient:0x00007f6698176f78 @client=#<RedisClient redis://10.0.0.2:6379/0>, @_pipeline_class=Sidekiq::RedisClientAdapter::CompatClient::Pipeline>], @max=5, @mutex=#<Thread::Mutex:0x00007f66981700d8>, @resource=#<Thread::ConditionVariable:0x00007f6698170060>, @shutdown_block=nil>, @key=:"pool-44800", @key_count=:"pool-44800-count">>, @queues=["mailers", "critical", "default", "low"], @concurrency=5, @redis=#<ConnectionPool:0x00007f669818e0d8 @size=5, @timeout=1, @available=#<ConnectionPool::TimedStack:0x00007f669818dfe8 @create_block=#<Proc:0x00007f669818e038 /var/deploy/myapp/web_head/shared/bundle/ruby/3.1.0/gems/sidekiq-7.0.0/lib/sidekiq/redis_connection.rb:22>, @created=0, @que=[], @max=5, @mutex=#<Thread::Mutex:0x00007f669818dc78>, @resource=#<Thread::ConditionVariable:0x00007f669818dc50>, @shutdown_block=nil>, @key=:"pool-44820", @key_count=:"pool-44820-count">>

I think it's from this but I haven't checked deeply. With the 6.x version works without problems

deanpcmad commented 2 years ago

I didn't realise Sidekiq 7 had released but I'll take a look. Can you share your sidekiq.yml?

xdmx commented 2 years ago

Sure, it's a pretty simple one:

---
:concurrency: 5
:dynamic: true
:queues:
  - mailers
  - critical
  - default
  - low

And then I have a limit on dynamic queues in the sidekiq.rb:

Sidekiq.configure_server do |config|
  config[:limits] = Hash.new do |_, queue_name|
    queue_name.to_s.starts_with?("proj:") ? 1 : nil
  end
end
andreasunterhuber commented 1 year ago

Any update on this issue? @xdmx

deanpcmad commented 1 year ago

I'm afraid I've not had chance to properly look into this yet but if anyone finds a fix, feel free to create a PR

jlestavel commented 1 year ago

Hi everyone, I did propose some possible fix in #147 and #150. I didn't know much about the sidekiq-limit_fetch implementation, so I can't guarantee that the fix is right and optimal 😬 !

deanpcmad commented 1 year ago

If @xdmx and @andreasunterhuber can test PR #150 fixes this, that would be great :)

deanpcmad commented 1 year ago

I've just pushed v4.4.0 which should fix this :)