arturictus / sidekiq_alive

Liveness probe for Sidekiq in Kubernetes deployments
MIT License
188 stars 57 forks source link

`RedisClient::CannotConnectError: stream closed in another thread` after updates #89

Closed fwolfst closed 1 year ago

fwolfst commented 1 year ago

Thanks for the work on sidekiq-alive. After a recent update (to 2.2 and sidekiq 7.1), I see following errors on a multi-instance installation regularly (not reproducible, 1-x times per day under no load). There is no further information on the Job itself (its in the proper instance-specific queue, and of class SidekiqAlive::Worker.

image

How can we further debug this?

WoutDev commented 1 year ago

I see a similar error in my death handler. It seems to (timewise) roughly match a k8s rolling update on my Sidekiq pod. I only have a single pod running though.

juanbandajr commented 1 year ago

I've been seeing the same error a couple times a day. The errors stopped popping up after re-writing the redis calls to be contained within the sidekiq redis block.

module SidekiqAlive
  module Redis
    # Wrapper for `redis-client` gem used by `sidekiq` > 7
    # https://github.com/redis-rb/redis-client
    class RedisClientGem < Base
      def set(key, time:, ex:)
        Sidekiq.redis { |redis| redis.call('SET', key, time, ex: ex) }
      end

      def get(key)
        Sidekiq.redis { |redis| redis.call('GET', key) }
      end

      def match(key)
        Sidekiq.redis { |redis| redis.scan('MATCH', key).map { |key| key } }
      end

      def delete(key)
        Sidekiq.redis { |redis| redis.call('DEL', key) }
      end
    end
  end
end

Sidekiq's redis adapter catches certain connectivity errors and reconnects, as long as the calls are performed within the block: https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/capsule.rb#L97-L117

fwolfst commented 1 year ago

@juanbandajr looks good, allthough understanding the root-cause could be important. @arturictus any opinion? Should we open a PR?

arturictus commented 1 year ago

Thanks, @juanbandajr. @fwolfst checking the Sidekiq code makes all the sense to run the commands inside the block. https://github.com/sidekiq/sidekiq/blob/0175cdbe03c3a9edf908a155ae6f006061b5c224/lib/sidekiq/config.rb#L155-L174 Would you like to make a PR?

fwolfst commented 1 year ago

Thanks, @juanbandajr. @fwolfst checking the Sidekiq code makes all the sense to run the commands inside the block. https://github.com/sidekiq/sidekiq/blob/0175cdbe03c3a9edf908a155ae6f006061b5c224/lib/sidekiq/config.rb#L155-L174 Would you like to make a PR?

Sorry, too late ... :)

arturictus commented 1 year ago

solved