search

brandonhilkert / sucker_punch

Sucker Punch is a Ruby asynchronous processing library using concurrent-ruby, heavily influenced by Sidekiq and girl_friday.
MIT License
2.65k stars 114 forks source link

sucker_punch with Passenger, how does the workers count works and how to check the stats on running jobs? #169

Closed dashbitla closed 8 years ago

dashbitla commented 8 years ago

I have Rails application running with Passenger to run 10 Processes and sucker_punch Jobs are defined to have 5 workers each for 3 jobs (5 * 3 = 15 total workers).

From what I read - sucker_punch runs in a Process. In Passenger environment, do these workers span for each Passenger Process or is it for the entire Passenger environment?

If its per Passenger Process then it means 15 workers * 10 processes = 150 total workers. In this case database pool size should be 150 + 5 (default pool size say) = 155?? It means MySQL max_connections should be at least 155.

If its for the entire Passenger environment, then 15 workers * 1 Passenger environment = 15 total workers. In this case database pool size should be 15 + 5 (default pool size say) = 20?? It means MySQL max_connections should be at least 20.

Is that right??

Also, if we have multiple servers running Passenger in 3 different servers, the the above will become 3x - right?

Is there a way to report the statistics of sucker_punch running jobs, stuck jobs and the jobs in queue someway? How would we report this across multiple servers??

Thank you!

This Gem is really awesome - as we plan to use in AWS beanstalk, we don't necessarily need to spin of worker environments, as this can do the job perfectly!

brandonhilkert commented 8 years ago

They do not span processes - so first question would need 150 connections in the pool at least. Not including connections needed for the web application.

If you have multiple servers, it would be 3x, yup!

The stats will be per process, you can call Queue.stats

dashbitla commented 8 years ago

Thank you!

Queue.stats => so this will give the stats related to one Process from the Passenger Process thats requesting this data right?

If we need to report it for the complete Passenger with 10 processes environment, is there a way to show the aggregated queue starts across all the processes?

brandonhilkert commented 8 years ago

That's right, it's per-process. There's currently no way to report across process.

http://brandonhilkert.com

On Tue, Mar 15, 2016 at 5:40 PM, Dash Bitla notifications@github.com wrote:

Thank you!

Queue.stats => so this will give the stats related to one Process from the Passenger Process thats requesting this data right?

If we need to report it for the complete Passenger with 10 processes environment, is there a way to show the aggregated queue starts across all the processes?

— You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub https://github.com/brandonhilkert/sucker_punch/issues/169#issuecomment-197034184

dashbitla commented 8 years ago

OK Got it. Thank you for all the clarifications.