Start to think about Celery and RP

[andre-merzky]

What is the context of this issue? Naively, Celery could be considered an executor for RP - but what would be the motivation / advantages to look into this? In the MolSSI context, we could also plug Celery behind the same RP REST API, FWIW - is that what this is about?

[dgasmith]

From discussions on our end, we could look at implement Celery in two different ways:

• Have a Celery Broker on QCFractal that RP would connect and have multiple jobs pushed to it.
• Take AMPQ standard as the REST API.

The first doesn't seem to be a use case that Celery really looks at unfortunately. I have not really seen a way that we can treat workers as "bulk workers" that would execute many tasks simultaneously. We would need to prepackage them into blocks on our end I believe. It also seems that Celery very much wants the broker and worker to effectively be the same base program, perhaps it would be better to strip it to more bare bones such as using their Kombu communicator.

For the second this could be more appropriate, but again I don't quite see an easy way to bundle multiple jobs with AMPQ.

I very much like the idea of using something with the size and integrity of Celery, but cannot quite figure out how to make it work.

[andre-merzky]

AMQP as in Asynchronous Message Queue Protocol? If so, I am not sure how that would relate to Celery and REST, really. If AMPQ, then what does that stand for?

I very much like the idea of using something with the size and integrity of Celery, but cannot quite figure out how to make it work.

I really understand that feeling, but it does sound somewhat like a solution looking for a problem ;-) It probably would make sense to detail the use case and problem space for this, and only then look into how it can be implemented? What exact capabilities makes Celery interesting for MolSSI?

[vivek-bala]

It also seems that Celery very much wants the broker and worker to effectively be the same base program, perhaps it would be better to strip it to more bare bones such as using their Kombu communicator.

I share that assessment as well. If the requirement is simply to communicate 'task' objects between different end points, we should definitely consider using just the communicator. Kombu might be a suitable choice as it supports several backends (in addition to those using AMQP) including redis and mongo dbs that you might already be considering for your data stores.

[dgasmith]

@andre-merzky Advanced Message Queuing Protocol. Mostly looking around to see if someone has already implemented many of the features we are looking for and it would be best to adhere to standards where possible. @doaa-altarawy do you have additional thoughts there?

@vivek-bala I think that is probably what we are looking for currently. I would be curious to share notes on that package at some point.

[andre-merzky]

already implemented many of the features we are looking for

What are those features?

[dgasmith]

Worker heartbeats, periodic tasks, task fusion, task routing, multi-queue support, high level task management with GUI front ends, automatic restart capabilities, full specifications of where a job was executed, standard communication format, prefetching. Not sure there are actual requirements there, but several features would be potentially useful if available.

Our own conclusion was that celery didn't match our requirements well, but it was an initial thought to explore the possibility. More to the point we were interested if we could have a Celery broker on our end and use that tech to communicate to RP. It does not appear to be the case.

[andre-merzky]

Thanks for that list, that gives me some useful context!