search

distributed-system-analysis / pbench

A benchmarking and performance analysis framework
http://distributed-system-analysis.github.io/pbench/
GNU General Public License v3.0
188 stars 108 forks source link

Consider the use of a queue for jobs to be done on the pbench server #1849

Open portante opened 4 years ago

portante commented 4 years ago

We might want to consider Kafka or python-rq (based on Redis) for replace the cronjobs on the pbench server side with workers that pull from the queues and service the jobs posted.

npalaska commented 4 years ago

Hey @portante can you explain this issue a little bit, do you mean all the cronjobs supposed to be in the queue structure or is regarding tracking jobs to be done on one tarfile after it gets uploaded?

dbutenhof commented 4 years ago

This refers to the use of Redis pub/sub for Tool Meister communication, in particular.

At some point when we microcomponent-ize the server and get rid of the simplistic crontab launch intervals, we'll also want to use messaging to coordinate activities; it might not be a bad idea to choose something now that we can grow into later. Redis pub/sub may be OK while things are simple, (and encapsulating our current inline Redis use into a class, either our own or a package like the one referenced, might help with maintainability); but RabbitMQ or Kafka may give us more options in the long run.

portante commented 4 years ago

Well, it was not my intention to refer to the Tool Meister, this is really about the pbench server cronjobs.