Closed fwolfst closed 1 year ago
I see a similar error in my death handler. It seems to (timewise) roughly match a k8s rolling update on my Sidekiq pod. I only have a single pod running though.
I've been seeing the same error a couple times a day. The errors stopped popping up after re-writing the redis calls to be contained within the sidekiq redis block.
module SidekiqAlive
module Redis
# Wrapper for `redis-client` gem used by `sidekiq` > 7
# https://github.com/redis-rb/redis-client
class RedisClientGem < Base
def set(key, time:, ex:)
Sidekiq.redis { |redis| redis.call('SET', key, time, ex: ex) }
end
def get(key)
Sidekiq.redis { |redis| redis.call('GET', key) }
end
def match(key)
Sidekiq.redis { |redis| redis.scan('MATCH', key).map { |key| key } }
end
def delete(key)
Sidekiq.redis { |redis| redis.call('DEL', key) }
end
end
end
end
Sidekiq's redis adapter catches certain connectivity errors and reconnects, as long as the calls are performed within the block: https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/capsule.rb#L97-L117
@juanbandajr looks good, allthough understanding the root-cause could be important. @arturictus any opinion? Should we open a PR?
Thanks, @juanbandajr. @fwolfst checking the Sidekiq code makes all the sense to run the commands inside the block. https://github.com/sidekiq/sidekiq/blob/0175cdbe03c3a9edf908a155ae6f006061b5c224/lib/sidekiq/config.rb#L155-L174 Would you like to make a PR?
Thanks, @juanbandajr. @fwolfst checking the Sidekiq code makes all the sense to run the commands inside the block. https://github.com/sidekiq/sidekiq/blob/0175cdbe03c3a9edf908a155ae6f006061b5c224/lib/sidekiq/config.rb#L155-L174 Would you like to make a PR?
Sorry, too late ... :)
solved
Thanks for the work on sidekiq-alive. After a recent update (to 2.2 and sidekiq 7.1), I see following errors on a multi-instance installation regularly (not reproducible, 1-x times per day under no load). There is no further information on the Job itself (its in the proper instance-specific queue, and of class
SidekiqAlive::Worker
.How can we further debug this?