NAMD / pypelinin

Python library to distribute jobs and pipelines among a cluster
3 stars 5 forks source link

Create a CLI to run daemons #44

Open turicas opened 11 years ago

turicas commented 11 years ago

This issue is actually related to lots of tasks and has some dependencies:

A sample config file should be like:

[global]
working_directory = /home/my_project/
api = tcp://127.0.0.1:5550
broadcast = tcp://127.0.0.1:5551
uid = my_user # optional, only if needs setuid
gid = my_group # optional, only if needs setgid

[router]
name = My Router
log_path = /tmp/router.log
pid_file = /tmp/router.pid

[pipeliner]
name = My Pipeliner
log_path = /tmp/pipeliner.log
pid_file = /tmp/pipeliner.pid

[broker]
name = My Broker
log_path = /tmp/broker.log
pid_file = /tmp/broker.pid
worker_module = workers
store_class = mymodule.MyStoreClass
number_of_workers = 40
store_custom_config_1 = my-value
store_custom_config_2 = other-value

All store_* directives inside broker section should be passed to Broker and then to StoreClass.__init__, as Broker should not ask for configuration to Router (as described in #56).

And the main usage should be like:

pypelinin -f pypelinin.conf <start|stop|restart> <all|router|pipeliner|broker>

Some command-line arguments are worth implementing:

A manual page for pypelinin command is also desired, but it's not really needed on the first implementation.

israelst commented 11 years ago

Good shot!

turicas commented 11 years ago

Would be nice if we are able to specify a cluster shared cryptography key (for authentication of the daemons) in this config file too (but #11 needs to be implemented before it).

turicas commented 11 years ago

Currently, the "daemons" are not real daemons since they do not detach from tty, neither create new process session id and other things daemons should do by definition (read more about UNIX daemons at Wikipedia). We should use the library python-daemon (described in PEP-3143) to implement it (it's very easy to use).

turicas commented 11 years ago

Since there are lots of tasks involved in this, I've created other issues (as dependencies of this one) and updated its description with dependencies and describing some possible command-line options.

israelst commented 11 years ago

We could use something like number_of_workers_per_cpu instead of number_of_workers in order to have more flexibility