jondot / sneakers

A fast background processing framework for Ruby and RabbitMQ
https://github.com/jondot/sneakers
MIT License
2.24k stars 333 forks source link

Stops "ack!"ing after 10 messages #465

Closed alilland closed 1 year ago

alilland commented 1 year ago

Im having some strange behavior,

I installed sneakers on a ruby app, I have about 10 other apps with the same code which are not experiencing this issue

an ack! should be sent under all circumstances, but after around 10 messages it appears my worker is not even receiving anything to work off. The message just sits "ready" forever

Screen Shot 2022-10-26 at 5 04 00 PM
## Boot Script
Sneakers.configure(
  heartbeat: 60,
  amqp: "amqp://#{ENV['RABBITMQ_USER']}:#{ENV['RABBITMQ_PASS']}@#{ENV['RABBITMQ_HOST']}",
  daemonize: ENV['RACK_ENV'] == 'production',
  pid_path: './sneakers.pid',
  vhost: '/',
  exchange: 'sneakers',
  exchange_type: :direct,
  timeout_job_after: 2.minutes,
  workers: 1,
  properties: {
    connection_name: 'worker'
  }
)
Sneakers.logger.level = Logger::DEBUG
workers = Sneakers::Runner.new([Workers::BB])
workers.run
## Worker
module Workers
  ##
  class BB
    include Sneakers::Worker

    from_queue :backbone

    def work(msg)
      start_time = Time.now
      status = 'green'
      log :debug, 'start'

      obj = JSON.parse(msg)
      log :debug, "#{obj['cmd']} ... start"

      case obj['cmd'].to_sym
      when :EVENT_INTERFACE
        BACKBONE::EventInterface.process_received(obj['payload'])
      end
      ack!
      log :info, "finished #{status}, took #{Time.now - start_time}"
    rescue StandardError => e
      status = 'red'
      log :fatal, e.message
      log :fatal, e.backtrace
      ack!
      log :fatal, "finished #{status}, took #{Time.now - start_time}"
    end
  end
end
michaelklishin commented 1 year ago

According to the screenshot, there are 10 delivered but unacknowledged messages. Most likely the prefetch value used is 10 and no deliveries were acknowledged.

That's the best hypothesis I have without a traffic capture and RabbitMQ logs.

llxff commented 1 year ago

Could it be the problem that ack! should be the last call in the work method? Because ack! just returns a symbol that is used later to decide if the processing was successful 🤔

https://github.com/jondot/sneakers/blob/v2.12.0/lib/sneakers/worker.rb#L33 https://github.com/jondot/sneakers/blob/v2.12.0/lib/sneakers/worker.rb#L84