search

redis-rb / redis-client

Simple low level client for Redis 6+
MIT License
124 stars 60 forks source link

RedisClient::ConnectionError Broken pipe #116

Closed wnm closed 1 year ago

wnm commented 1 year ago

We exchanged our old redis setup:

@pool = ConnectionPool.new(size: 5, timeout: 5) { Redis.new(url: Settings.redis.url, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }) }

with redis-client this morning:

redis_config = RedisClient.config(url: Settings.redis.url, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })
    @pool = redis_config.new_pool(timeout: 5, size: 5)

Most things work fine, but we got a few RedisClient::ConnectionError Broken pipe errors since upgrading. Any pointers on how to debug this, and what the reason could be?

byroot commented 1 year ago

Could you share the full backtrace for these errors?

wnm commented 1 year ago 
 File /app/app/jobs/email/imap/concerns/lock_manager.rb:64 in lock
File /app/app/jobs/email/imap/concerns/lock_manager.rb:29 in block (2 levels) in <module:LockManager>
Hide 96 non-project frames
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:467 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:467 in block in make_lambda
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:199 in block (2 levels) in halting
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:687 in block (2 levels) in default_terminator
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:686 in catch
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:686 in block in default_terminator
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:200 in block in halting
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:595 in block in invoke_before
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:595 in each
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:595 in invoke_before
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:116 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/active_job.rb:67 in perform
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/active_job.rb:23 in block (4 levels) in <main>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/i18n-1.13.0/lib/i18n.rb:322 in with_locale
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/translation.rb:9 in block (2 levels) in <module:Translation>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/core_ext/time/zones.rb:66 in use_zone
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/timezones.rb:9 in block (2 levels) in <module:Timezones>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:138 in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:58 in _perform_job
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:20 in _perform_job
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:46 in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:14 in block in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:25 in block in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/notifications.rb:206 in block in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/notifications/instrumenter.rb:24 in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/notifications.rb:206 in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:35 in instrument
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/instrumentation.rb:14 in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/logging.rb:18 in block in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/tagged_logging.rb:99 in block in tagged
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/tagged_logging.rb:37 in tagged
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/tagged_logging.rb:99 in tagged
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/logging.rb:25 in tag_logger
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/logging.rb:18 in perform_now
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:24 in block in execute
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:118 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/railtie.rb:54 in block (4 levels) in <class:Railtie>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:92 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:72 in block in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:88 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:71 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/railtie.rb:53 in block (3 levels) in <class:Railtie>
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in instance_exec
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:127 in block in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/callbacks.rb:138 in run_callbacks
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/execution.rb:22 in execute
File /app/vendor/bundle/ruby/3.2.0/gems/activejob-7.0.4.3/lib/active_job/queue_adapters/sidekiq_adapter.rb:42 in perform
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:209 in execute_job
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:177 in block (2 levels) in process
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:180 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:183 in block in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/rollbar-3.4.0/lib/rollbar/plugins/sidekiq/plugin.rb:11 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:182 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:183 in block in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/sidekiq/server.rb:25 in block in call
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:384 in perform_action_with_newrelic_trace
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/instrumentation/sidekiq/server.rb:20 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:182 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:183 in block in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/metrics/tracking.rb:26 in track
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/metrics/tracking.rb:122 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:182 in traverse
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/middleware/chain.rb:173 in invoke
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:176 in block in process
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:140 in block (6 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_retry.rb:114 in local
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:139 in block (5 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/rails.rb:16 in block in call
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:92 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:72 in block in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/execution_wrapper.rb:92 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.4.3/lib/active_support/reloader.rb:71 in wrap
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/rails.rb:15 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:135 in block (4 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:270 in stats
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:130 in block (3 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_logger.rb:13 in call
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:129 in block (2 levels) in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_retry.rb:81 in global
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:128 in block in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/job_logger.rb:39 in prepare
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:127 in dispatch
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:175 in process
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:82 in process_one
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/processor.rb:72 in run
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/component.rb:10 in watchdog
File /app/vendor/bundle/ruby/3.2.0/gems/sidekiq-7.1.0/lib/sidekiq/component.rb:19 in block in safe_thread
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/tracer.rb:434 in block (2 levels) in thread_block_with_current_transaction
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/tracer.rb:356 in capture_segment_error
File /app/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.2.2/lib/new_relic/agent/tracer.rb:433 in block in thread_block_with_current_transaction
RedisClient::ConnectionError: Broken pipe (Most recent call first)
Hide 28 non-project frames
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:76 in rescue in write
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:73 in write
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/connection_mixin.rb:30 in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:211 in block (2 levels) in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/middlewares.rb:16 in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:210 in block in call
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:626 in ensure_connected
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client.rb:209 in call
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:172 in block (2 levels) in lock
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/pooled.rb:30 in block in with
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:107 in block (2 levels) in with
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:106 in handle_interrupt
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:106 in block in with
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:103 in handle_interrupt
File /app/vendor/bundle/ruby/3.2.0/gems/connection_pool-2.4.0/lib/connection_pool.rb:103 in with
File /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.14.1/lib/redis_client/pooled.rb:26 in with
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:171 in block in lock
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:216 in recover_from_script_flush
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:170 in lock
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:266 in block (2 levels) in lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:266 in select
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:266 in block in lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:319 in timed
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:265 in lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:240 in block in try_lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:236 in times
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:236 in try_lock_instances
File /app/vendor/bundle/ruby/3.2.0/gems/redlock-2.0.1/lib/redlock/client.rb:69 in lock
File /app/app/jobs/email/imap/concerns/lock_manager.rb:64 in lock
File /app/app/jobs/email/imap/concerns/lock_manager.rb:29 in block (2 levels) in <module:LockManager>
byroot commented 1 year ago

Ok, I suspect the connection is lost for some reason, but that's probably not new.

See https://github.com/redis-rb/redis-client#reconnection

redis-rb does retry by default, and that's probably what was hiding these events to you. You can pass reconnect_attempts: 1 to restore the old behavior.

wnm commented 1 year ago

ah fantastic, thanks so much for the quick reply!!! ❤️