search

timofurrer / minion-ci

minimalist, decentralized, flexible Continuous Integration Server for hackers.
MIT License
43 stars 4 forks source link

Consider using celery for task distribution #14

Open timofurrer opened 7 years ago

timofurrer commented 7 years ago

We should consider celery for task distribution instead of multiprocessing.

This would bring some convenience functions but is another third party library minion-ci would depend on.. And with celery we would need some setup backends, like rabbitmq and/or redis.

ghost commented 7 years ago

I am not completely sure, but according to this we could use mongodb as a backend.

Of course mongodb isn't a pre for celery it could be an option. I think it would be an awesome feature if one could choose between native multiprocessing or between installing an extra dependency.

Maybe we can create something like a config file for the server?

timofurrer commented 7 years ago

I am not completely sure, but according to this we could use mongodb as a backend.

Oh yeah, I completely forgot about this .. however, a database a communication backend is not really our best option - In our case feasible though.

I'd might be pretty ugly to implement both: multiprocessing and celery. The interface is not really the same.

ghost commented 7 years ago

True, I think we should stay with multiprocessing. It is native python, fast enough and the code is readable.

I can do some work on integrating with celery and than we can check the differences? If it is a huge performance change or so we should consider it.