A way to read the status of a job

[djay]

some queues allow you to query the status of a job. For example it would be useful if had a webpage that used some task id and gave feedback if the task was queued/running/completed. Some queues like celery allow for custom feedback messaging.

[datakurre]

It would be possible (at least with the currently supported "local" and "Redis" queues), but I'd need some help to design, what information should be available, how should it be made visible and to whom and when it could be deleted.

Technically

• each task would be given a uuid
• each local queue would have simple dictionary to store task related data
• each redis queue would simply use redis to save task related data

And then there should be a persistent logging utility to store some metadata and return codes for each tasks. There could be on/off policy for keeping the log and some UI to clean it.

[datakurre]

This could get easily too complex, so what could be "minimal viable product" here? :)

[djay]

looking at p.a.async (http://plone.org/products/plone.app.async) they give pending-status + result status and also a callback to get the result. In this case, I'm not sure if there is value in returning a result (would it be the response body?). I'd say the MVP would knowing if the task was pending given it's UID, and perhaps knowing it's final response code ie 200, 500? I guess minimal minimal would be know if its pending, running or (not in queue, ie done). Then you don't need to deal with how long to keep the data.

If you look at appengine, the tasks don't return results but have methods such as eta which indicate if it's running, or how long it will take to run. https://developers.google.com/appengine/docs/python/taskqueue/tasks but the queue seems to be able to get the result or register a callback https://developers.google.com/appengine/docs/python/taskqueue/rpcclass#get_result. I think what that means is you can only get the result if you register a callback or synchronise yourself to wait for a result

[djay]

I think based on the above the min would be something get to pending/running/position in queue? This would get you the ability to have feedback on the page saying "still processing" etc.

After that, an api to register a callback or blocking call to get the result if you want it. This would allow you to farm out some computationally complex tasks to multiple different servers and hold up the request until all had finished for example. I don't think the callback needs to support calling back in a new transaction. To do that the programmer could just create a new task. Instead perhaps the callback is something that only lasts during the current transaction/request. For example you could farm out a task, and register a callback for each but if you're transaction ends your callback is automatically unregistered and you won't get notified of a result.

Personally I only have a usecase currently for part A but I can see why other apis have part B.

[datakurre]

I'm still thinking, if the implementation should rely on the used queue or not (volatile instance local / redis / zc.queue [in the future]).

I mean, when asked about a state of task, should that be always looked directly from the queue. An alternative would be to keep a log of dispatched and handled messages in ZODB (as a local utility for the current site). The latter would give a nice history / statistics for the tasks, but would miss tasks that are handled outside Plone (but that's not supported out-of-the box anyway).

A kind of callback could be given as an event inside Plone. (c.taskqueue could fire a custom event after a task has been successfully executed, but before it has been committed; technically an event within PubSuccess-event).

[djay]

The more I think about it the only callback or result that makes sense is one where the producer is blocking while waiting for a result. Anything else you can handle yourself by saving data in your task, or making your own event.

[datakurre]

If I understood correctly, this would not be possible as long as tasks are queued only after a successful transactions, which is the current implementation.

Couldn't this kind of user cases be controlled by workflows?

1. Object's state is changed to "computing" and distributed computing tasks are queued after successful transaction.
2. Each task is executed separately, and the result is saved into the object (or returned in a separate message and saved from that).
3. The last task changes the state to 'completed' to mark that all the distributed tasks are done.

Of course, this would cause separate transaction for each result, but workflow could have 'abort' transition, which would return the object into its previous state before the tasks.