search

hamster-dev / hamster-core

API / Celery backend
MIT License
2 stars 1 forks source link

Worker communication with core? #4

Open mikewaters opened 10 years ago

mikewaters commented 10 years ago

How do you anticipate the workers will communicate job status with the rest of the system? Will there be a world-accessible database, or will workers use AMQPBackend, or something else?

gregarmer commented 10 years ago

Nothing set in concrete yet, but I figured we could push the worker output back to the database using the standard celery result stuff, once the job is completed. That output can be saved with the rest of the build data to whichever DB is configured in Django, and will be accessible through the API to the UI (or anything else using the API).

The only unknown is getting realtime output from the worker back to the browser (as per #2).

mikewaters commented 10 years ago

My question is really about remote workers, sorry I wasn't clear. Typically I don't make the database available to the WAN, it is only accessible by my app servers.
If you want a user to be able to spawn a remote worker (for instance, your laptop running mac builds - one of your initial use cases) we need to consider how it's going to communicate with the master securely. Using ssh for a bidirectional master/slave communication channel is Jenkins' solution, but we can instead one of celery's other result backends since a dedicated channel doesn't make sense for a disconnected/async model.

gregarmer commented 10 years ago

Oh, I get what you're asking now - sorry.

I guess an option would be to run a local worker on the server that will process build results (and probably a bunch of other periodic maintenance tasks) that persists build results to the database, then have the build workers push the results back over AMQP into a separate queue for that purpose.

We already know the worker can connect over AMQP since it got the build job, so this keeps the requirements simple for workers (just AMQP).