Closed steakchaser closed 5 years ago
Oh. Interesting. I will check ASAP
@ixti quick update - I've actually not been able to replicate this locally or in my staging environment 😬. One interesting thing to note is that when this was in prod, there was a period where we were queuing new jobs with sidekiq 5.1.3, but then processing them with 5.0.5 - also, tried this in staging and it seemed to work fine.
Update - seeing the same issue with matching Sidekiq client + server versions (5.0.5 or 5.1.3). Have still only been able to reproduce this issue in production, under load. See also mperham/sidekiq/issues/3887.
Can you try to upgrade json gem? Also did you (by any chance) monkey-patched Sidekiq::Throttled::Fetch? It seems like brpop
returns one element instead of tuple of queue and payload. You can check what's going one like this:
class Sidekiq::Throttled::Fetch
module Debug
def retrieve_work
super
rescue
warn "last brpop was: #{@last_work.inspect}" if defined? @last_work
raise
end
def brpop
super.tap { |work| @last_work = work }
end
end
prepend Debug
end
Recently attempted a Sidekiq upgrade from 5.0.5 -> 5.1.3 and encountered hundreds of errors originating from
sidekiq-throttled
. Reverting back resolved the issue.Error message:
Example Worker class: