search

ruby-shoryuken / shoryuken

A super efficient Amazon SQS thread based message processor for Ruby. This project is in MAINTENANCE MODE. Reach me out on Slack link on the description if you want to become a new maintainer.
Other
2.06k stars 280 forks source link

Potential issues with polling and receive messages #771

Closed VarunNatraaj closed 4 months ago

VarunNatraaj commented 4 months ago

Hello,

First off, love the library. Big fan and user from several years ago.

On the issue, we had a lot of messages stagnated in our SQS queue, and upon investigating, we noticed that the in flight messages seem to be in lower numbers than usual. The only recent update that we did from our side seems to be to the AWS SQS client and we haven't faced any issues until the today. We run 25 tasks / processes with concurrency 25 and batch poll (10), so we usually see messages in flight anywhere between 3000-6000, but today we're seeing 250-300.

Just wanted to check if there are any known issues with latest gem and latest AWS SQS client? I'm currently on these libs.

shoryuken (6.2.1)
aws-sdk-sqs (1.80.0)
      aws-sdk-core (~> 3, >= 3.201.0)
      aws-sigv4 (~> 1.5)
aws-sdk-core (3.201.1)

Also upgraded and downloaded to versions mentioned in this issue #766 but no avail. Anyone else facing similar issues that can help us out?

VarunNatraaj commented 4 months ago

Also noticing errors like these frequently off late in the logs

2024-07-13T06:22:07Z 49 TID-1yf0 ERROR: /usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:126:in `block in wait_poll'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:115:in `loop'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:115:in `wait_poll'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:76:in `internal_poll'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:202:in `internal_poll'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:71:in `block in poll'
/usr/local/lib/ruby/3.1.0/monitor.rb:202:in `synchronize'
/usr/local/lib/ruby/3.1.0/monitor.rb:202:in `mon_synchronize'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:80:in `synchronize'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool/queue.rb:71:in `poll'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:644:in `acquire_connection'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:341:in `checkout'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:181:in `connection'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_handler.rb:211:in `retrieve_connection'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_handling.rb:313:in `retrieve_connection'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/connection_handling.rb:280:in `connection'
/usr/local/bundle/gems/activerecord-7.0.4/lib/active_record/core.rb:433:in `cached_find_by_statement'
/usr/local/bundle/gems/composite_primary_keys-14.0.4/lib/composite_primary_keys/core.rb:37:in `find'
/app/app/workers/calendar_workers/microsoft/event_sync_worker.rb:38:in `process_message'
/app/app/workers/calendar_workers/microsoft/event_sync_worker.rb:27:in `block in perform'
/app/app/workers/calendar_workers/microsoft/event_sync_worker.rb:26:in `each'
/app/app/workers/calendar_workers/microsoft/event_sync_worker.rb:26:in `perform'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/processor.rb:21:in `block (3 levels) in process'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:98:in `block in invoke'
/app/lib/middleware/shoryuken_logger.rb:21:in `block in call'
/usr/local/bundle/gems/activesupport-7.0.4/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/usr/local/bundle/gems/activesupport-7.0.4/lib/active_support/tagged_logging.rb:37:in `tagged'
/usr/local/bundle/gems/activesupport-7.0.4/lib/active_support/tagged_logging.rb:99:in `tagged'
/app/lib/middleware/shoryuken_logger.rb:20:in `call'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/server/active_record.rb:6:in `call'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/server/auto_extend_visibility.rb:10:in `call'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/server/auto_delete.rb:6:in `call'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/server/exponential_backoff_retry.rb:8:in `call'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/server/timing.rb:12:in `call'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:100:in `block in invoke'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/middleware/chain.rb:103:in `invoke'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/processor.rb:20:in `block (2 levels) in process'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/logging.rb:20:in `with_context'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/processor.rb:19:in `block in process'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/processor.rb:31:in `process'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/processor.rb:8:in `process'
/usr/local/bundle/gems/shoryuken-6.2.1/lib/shoryuken/manager.rb:100:in `block in assign'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24:in `block in execute'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:48:in `block in synchronize'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:48:in `synchronize'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:48:in `synchronize'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:22:in `execute'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/promise.rb:564:in `block in realize'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:359:in `run_task'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:350:in `block (3 levels) in create_worker'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:341:in `loop'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:341:in `block (2 levels) in create_worker'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:340:in `catch'
/usr/local/bundle/gems/concurrent-ruby-1.3.3/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:340:in `block in create_worker'