search

jondot / sneakers

A fast background processing framework for Ruby and RabbitMQ
https://github.com/jondot/sneakers
MIT License
2.24k stars 333 forks source link

How to handle Memory leak #397

Closed eldadvcita closed 5 years ago

eldadvcita commented 5 years ago

Hi, Unfortunately, we maintain several ancient projects which rely heavily on Sneakers. While profiling those projects, I noticed that their memory footprint increases over time.

In order to not deal with those projects, I need to come up with a workarounde. a common practice (php-fpm, puma using puma_worker_killer, unicorn using unicorn-worker-killer, etc) is to restart the workers based on (1) max number of requests or (2) max memory limit and I wonder if there is an equivalent feature build into Sneakers?

Please advice, Thanks!

michaelklishin commented 5 years ago

We don't have enough information to suggest something specific. A high number of concurrently processed deliveries can be one contributing factor. You should profile the process, otherwise we can only guess what really consumes the heap, and guessing is very inefficient.

eldadvcita commented 5 years ago

@michaelklishin, I'm not interested in fixing the situation as the process fail to release ROR/ActiveRecord objects from the heap due to different bugs in old/deprecated libraries. Therefore I'm looking for a simple solution which will simply kill a given worker once it reaches a certain memory limit or handled X amount of messages.

michaelklishin commented 5 years ago

Sneakers doesn't have any provisions for that. This is a monitoring problem and it should be addressed using suitable monitoring tools.