Concurrency misbehaving and causing duplicate jobs in queue

[danishsatkut]

I have been trying to get sidekiq-throttled (v1.0.0.alpha.1) to work with a new project (Rails v7.0.6 and Sidekiq v7.1.2) but I have been running into issues. When the sidekiq processing starts, the concurrency config is not respected. Once the first batch of 5 jobs (default sidekiq concurrency) are finished, the concurrency config starts working, but now it results in duplicate jobs (similar to #152).

I have attached the SampleJob and sidekiq.log which demonstrates the issue here: https://gist.github.com/danishsatkut/6a11d06eae2f7b464d285002856ba9b5

[danishsatkut]

So after playing around with different refs of the gem, I think the issue started somewhere after e5ac585. At that version, I wasn't seeing any duplication but the concurrency issue was still the same as I had encountered before. https://gist.github.com/danishsatkut/1c43401588f45368569499c3eae2982b

BTW I also downgraded sidekiq to v7.1.1

[emmahsax]

My team is using version 0.17.0, and we're seeing duplicate jobs as well.

[danishsatkut]

@emmahsax Which version of sidekiq are you using?

[emmahsax]

0.17.0. But I realized I didn't finish the docs. The README says the default lock is for 15 minutes, and that wasn't enough for our jobs. This is what I get for working at 5:30 in the morning 🤦🏼‍♀️ .

[ixti]

That's weird. Will debug.

[pentagon]

the same issue here, alpha seems not doing any concurrency management/throttling with the setup similar to the topic starter's one.

[ixti]

I have refactored the way BasicFetch is get patched. Should be good in >= v.1.0.0

[ixti]

This is the script I used to check:

require "sidekiq"
require "sidekiq/throttled"

puts "Sidekiq::VERSION            => #{Sidekiq::VERSION}"
puts "Sidekiq::Throttled::VERSION => #{Sidekiq::Throttled::VERSION}"

Sidekiq::Throttled.setup!

class SampleJob
  include Sidekiq::Job
  include Sidekiq::Throttled::Job

  sidekiq_throttle concurrency: { limit: 1 }

  def perform(id)
    puts "Processing ##{id}"
    sleep(10)
    puts "Done ##{id}"
  end
end

100.times { SampleJob.perform_async(_1.succ) }

Output

``` text > be sidekiq -r ./app.rb Sidekiq::VERSION => 7.2.0 Sidekiq::Throttled::VERSION => 1.0.1 2023-11-20T16:58:51.483Z pid=3049 tid=2f5 INFO: Sidekiq 7.2.0 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>nil} m, `$b .ss, $$: .,d$ `$$P,d$P' .,md$P"' ,$$$$$b/md$$$P^' .d$$$$$$/$$$P' $$^' `"/$$$' ____ _ _ _ _ $: ',$$: / ___|(_) __| | ___| | _(_) __ _ `b :$$ \___ \| |/ _` |/ _ \ |/ / |/ _` | $$: ___) | | (_| | __/ <| | (_| | $$ |____/|_|\__,_|\___|_|\_\_|\__, | .d$$ |_| 2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Running in ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] 2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: See LICENSE and the LGPL-3.0 for licensing details. 2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org 2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Sidekiq 7.2.0 connecting to Redis with options {:size=>5, :pool_name=>"default", :url=>nil} 2023-11-20T16:58:51.494Z pid=3049 tid=2f5 INFO: Starting processing, hit Ctrl-C to stop 2023-11-20T16:58:51.500Z pid=3049 tid=qd class=SampleJob jid=00cc61231cb117f81dd31853 INFO: start Processing #1 Done #1 2023-11-20T16:59:01.502Z pid=3049 tid=qd class=SampleJob jid=00cc61231cb117f81dd31853 elapsed=10.002 INFO: done 2023-11-20T16:59:01.511Z pid=3049 tid=pl class=SampleJob jid=48bec13f40aacc1fb3946018 INFO: start Processing #14 Done #14 2023-11-20T16:59:11.512Z pid=3049 tid=pl class=SampleJob jid=48bec13f40aacc1fb3946018 elapsed=10.002 INFO: done 2023-11-20T16:59:11.519Z pid=3049 tid=od class=SampleJob jid=3063d342d29da264b4b07ac3 INFO: start Processing #20 Done #20 2023-11-20T16:59:21.519Z pid=3049 tid=od class=SampleJob jid=3063d342d29da264b4b07ac3 elapsed=10.0 INFO: done 2023-11-20T16:59:23.520Z pid=3049 tid=pl class=SampleJob jid=68cc2c6c29f92560be4165e4 INFO: start Processing #37 Done #37 2023-11-20T16:59:33.521Z pid=3049 tid=pl class=SampleJob jid=68cc2c6c29f92560be4165e4 elapsed=10.001 INFO: done 2023-11-20T16:59:33.535Z pid=3049 tid=oh class=SampleJob jid=7823134f81d36c20ff8fef0c INFO: start Processing #54 Done #54 2023-11-20T16:59:43.536Z pid=3049 tid=oh class=SampleJob jid=7823134f81d36c20ff8fef0c elapsed=10.001 INFO: done 2023-11-20T16:59:43.545Z pid=3049 tid=qd class=SampleJob jid=7db02baafd2b97f58da996b2 INFO: start Processing #65 Done #65 2023-11-20T16:59:53.546Z pid=3049 tid=qd class=SampleJob jid=7db02baafd2b97f58da996b2 elapsed=10.001 INFO: done 2023-11-20T16:59:53.552Z pid=3049 tid=od class=SampleJob jid=019a13a53200fcfdf55e3c9b INFO: start Processing #71 Done #71 2023-11-20T17:00:03.553Z pid=3049 tid=od class=SampleJob jid=019a13a53200fcfdf55e3c9b elapsed=10.001 INFO: done 2023-11-20T17:00:05.534Z pid=3049 tid=n9 class=SampleJob jid=e4b16d0ffde0ebf5df5bc108 INFO: start Processing #82 Done #82 2023-11-20T17:00:15.535Z pid=3049 tid=n9 class=SampleJob jid=e4b16d0ffde0ebf5df5bc108 elapsed=10.001 INFO: done ... ```

[danishsatkut]

@ixti I will test and confirm if it works. Thank you for your efforts.

[danishsatkut]

@ixti I have tested and can confirm the issue is resolved after upgrading to v1.0.1 (https://github.com/ixti/sidekiq-throttled/pull/166)

[ixti]

@danishsatkut thank you!