search

redis-rb / redis-client

Simple low level client for Redis 6+
MIT License
124 stars 60 forks source link

Running into RedisClient::ConnectionError: stream closed in another thread #112

Closed fluke closed 1 year ago

fluke commented 1 year ago

Since I upgraded redis-client to 14.1 from 12.2 I have been seeing a large number of the above error. I am running this app on Heroku and using Heroku Redis. It is primarily used by redlock-rb gem which is there in the stacktrace.

Error is coming from: https://github.com/redis-rb/redis-client/blob/v0.14.1/lib/redis_client/ruby_connection.rb#L92

Stacktrace:

2023-04-27T05:10:28Z 21 TID-10v8 ERROR: /app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:101:in `rescue in read'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client/ruby_connection.rb:92:in `read'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client/connection_mixin.rb:31:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client.rb:211:in `block (2 levels) in call'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client/middlewares.rb:16:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client.rb:210:in `block in call'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client.rb:626:in `ensure_connected'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client.rb:209:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:172:in `block (2 levels) in lock'
/app/vendor/bundle/ruby/3.1.0/gems/redis-client-0.14.1/lib/redis_client.rb:182:in `with'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:171:in `block in lock'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:216:in `recover_from_script_flush'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:170:in `lock'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:266:in `block (2 levels) in lock_instances'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:266:in `select'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:266:in `block in lock_instances'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:319:in `timed'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:265:in `lock_instances'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:240:in `block in try_lock_instances'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:236:in `times'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:236:in `try_lock_instances'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:69:in `lock'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:104:in `lock!'
/app/app/lib/lock.rb:4:in `lock'
/app/app/models/intent.rb:132:in `deliver_notification'
/app/app/models/intent.rb:128:in `notify'
/app/app/workers/shopify/webhooks/inventory_levels/update_worker.rb:25:in `block in notify_intent'
/app/app/lib/lock.rb:5:in `block in lock'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:106:in `block in lock!'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:81:in `lock'
/app/vendor/bundle/ruby/3.1.0/gems/redlock-2.0.1/lib/redlock/client.rb:104:in `lock!'
/app/app/lib/lock.rb:4:in `lock'
/app/app/workers/shopify/webhooks/inventory_levels/update_worker.rb:23:in `notify_intent'
/app/app/workers/shopify/webhooks/inventory_levels/update_worker.rb:15:in `block in handle_webhook'
/app/app/workers/shopify/webhooks/inventory_levels/update_worker.rb:15:in `each'
/app/app/workers/shopify/webhooks/inventory_levels/update_worker.rb:15:in `handle_webhook'
/app/app/workers/shopify/webhooks/webhook_worker.rb:13:in `block in perform'
/app/vendor/bundle/ruby/3.1.0/gems/shopify_app-21.4.0/lib/shopify_app/session/session_storage.rb:14:in `block in with_shopify_session'
/app/vendor/bundle/ruby/3.1.0/gems/shopify_api-12.4.0/lib/shopify_api/auth/session.rb:81:in `temp'
/app/vendor/bundle/ruby/3.1.0/gems/sorbet-runtime-0.5.10676/lib/types/private/methods/call_validation.rb:256:in `bind_call'
/app/vendor/bundle/ruby/3.1.0/gems/sorbet-runtime-0.5.10676/lib/types/private/methods/call_validation.rb:256:in `validate_call'
/app/vendor/bundle/ruby/3.1.0/gems/sorbet-runtime-0.5.10676/lib/types/private/methods/call_validation.rb:177:in `block in create_validator_slow'
/app/vendor/bundle/ruby/3.1.0/gems/shopify_app-21.4.0/lib/shopify_app/session/session_storage.rb:13:in `with_shopify_session'
/app/app/workers/shopify/webhooks/webhook_worker.rb:12:in `perform'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/processor.rb:21:in `block (2 levels) in process'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:98:in `block in invoke'
/app/vendor/bundle/ruby/3.1.0/gems/bugsnag-6.25.2/lib/bugsnag/integrations/shoryuken.rb:29:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/server/active_record.rb:6:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/server/auto_extend_visibility.rb:10:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/server/auto_delete.rb:6:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/server/exponential_backoff_retry.rb:8:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/server/timing.rb:12:in `call'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/middleware/chain.rb:103:in `invoke'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/processor.rb:20:in `block in process'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/logging.rb:20:in `with_context'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/processor.rb:19:in `process'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/processor.rb:8:in `process'
/app/vendor/bundle/ruby/3.1.0/gems/shoryuken-5.3.2/lib/shoryuken/manager.rb:100:in `block in assign'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:48:in `block in synchronize'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:48:in `synchronize'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:48:in `synchronize'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:22:in `execute'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/promise.rb:564:in `block in realize'
/app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.16.0/lib/new_relic/agent/tracer.rb:423:in `block (2 levels) in thread_block_with_current_transaction'
/app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.16.0/lib/new_relic/agent/tracer.rb:356:in `capture_segment_error'
/app/vendor/bundle/ruby/3.1.0/gems/newrelic_rpm-8.16.0/lib/new_relic/agent/tracer.rb:422:in `block in thread_block_with_current_transaction'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
/app/vendor/bundle/ruby/3.1.0/gems/concurrent-ruby-1.2.0/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
byroot commented 1 year ago

Hi,

This isn't a bug in redis-client, it means you are using the same connection from multiple threads, you shouldn't.

In multi-threaded environment, you must either use a pool, or assign a single client per thread, it's all described in the readme.

Similar: https://github.com/leandromoreira/redlock-rb/issues/128

fluke commented 1 year ago

Thanks for the answer. I should have checked in the redlock issues as well, sorry.