search

ixti / sidekiq-throttled

Concurrency and rate-limit throttling for Sidekiq
MIT License
701 stars 75 forks source link

Throttle concurrency locks timing out, not detecting job end #66

Closed aaronjensen closed 1 year ago

aaronjensen commented 4 years ago

I haven't had a chance to dig into it yet, but today, our throttled jobs stopped processing. Rather, the concurrency locks each had to time out before allowing another job to go through. Things were processing fine, then there was a heroku-redis upgrade, then, hours later, our dynos restarted and throttled jobs stopped registering their completion. As far as I can tell, there were no code changes from when this worked to when it stopped working.

One possible thing of note is that we also use sidekiq-unique-jobs and this particular job was configured with both:

sidekiq_options queue: :medium, lock: :until_executed, on_conflict: :log, log_duplicate_payload: true
sidekiq_throttle concurrency: { limit: 5 }

I haven't had a chance to dig in and debug—for now I've removed the throttle and the jobs process normally.

Any hints on where to look? What might cause something like this? Could the redis upgrade be related?

Thank you!

Blokh commented 4 years ago

Hello, We're having the same issue - we're using strategy in order to control the concurrency of various jobs. e.g. Sidekiq::Throttled::Registry.add(:feed_listing_worker_concurrency_strategy, concurrency: { limit: 80, ttl: 2.hours.to_i })

When I am fetching the Zrange from the strategy key it shows me that it's full even tho there are no jobs running.

What might be the cause of this issue?

Thanks in advance, Daniel

ixti commented 4 years ago

Heya. Do you get any kind of errors? Like could it be possible that somehow process dies without calling Middlewares?

newbray commented 4 years ago

I have also experienced this issue using sidekiq-unique-jobs (without sidekiq-throttled). I think this is the problem for us: https://github.com/mhenrixon/sidekiq-unique-jobs/issues/486 We have disabled that gem for now

dchun commented 2 years ago

I have the same problem as well. Whenever heroku updates my redis server I get a redis timeout error. I don't use unique jobs so I don't think it's exclusive to unique jobs. Anyone figure this out?

ixti commented 1 year ago

I think it could be related to now removed IPC that was using pub/sub. If you have issues with 1.0.0.alpha - feel free to reopen this issue.