maki-nage / makinage

Stream Processing Made Easy
https://www.makinage.org
MIT License
38 stars 1 forks source link

add -j option for multi-processing #3

Open MainRo opened 3 years ago

MainRo commented 3 years ago

Multi-processing should be managed outside of maki-nage in production environment (via the deployment/orchestration tools). However on local usage, it would be useful to support forking in multiple processes instead of requiring the user to launch the application multiple times.

a -j [njobs] option should be added for this. njobs can have several values:

1 is a special value where no fork is done, and the master process is the single worker. In all other forking configurations, the master process will not act as a worker.

This requires a "fork" cyclotron driver that will be in charge of forking, waiting, and eventually killing the workers from the master (maybe this should be in cyclotron-std).