Consumer thread interrupt

[wojozer]

Hello amazing sneaker, Hope everything is OK with you! I ran into a problem where the process takes a long time when the consumer processes the messages. When the number of messages is too high, the consumer thread probability will break and retry without any exception. I have modified the timeout time without effect.

Strangely, I modified the number of workers, which defaults to 4, and I modified it to 2 and it no longer happens. What is the reason for this? I just can't understand it.

[wojozer]

The CPU is 4-core.

[michaelklishin]

That suggests that your CPU resources are insufficient for the consumer workload and four workers N threads is too much for them but two workers N threads is OK. The effective prefetch value is another factor that can potentially be relevant.

[wojozer]

Thanks @michaelklishin , How can I be sure that this is the cause? Because I don't see any exception logs occurring, and the consumer is retrying.

Also I see that the CPU monitor is always in a healthy state. threads is set to 1.

 WORKER_OPTIONS = {
     ack: true,
     threads: 1,
     prefetch: 1,
     heartbeat: 10,
     routing_key: routing_key,
     max_retries: 6,
     arguments: {
           'x-dead-letter-exchange' => "#{queue_name}.dlx",
           'x-dead-letter-routing-key' => routing_key,
     },
}.freeze

  Sneakers.configure  amqp: rabbitmq_url,
                      handler: SneakersHandlers::ExponentialBackoffHandler,
                      metrics: Sneakers::Metrics::NewrelicMetrics.new,
                      workers: 4,
......

Best Regards!

[michaelklishin]

Resource exhaustion may result in timeouts without any exceptions, or any exceptions that provide enough clues.