General discussion - reporting on workflow state

[di-sysops]

Hello fellow developers, this is not an issue per-se with the SlmQueue module specifically, but rather, a question about how one ought to report on the state of a given workflow.

For the sake of this discussion, let's assume we have just a single default queue.

We have a workflow consisting of the following three steps (based on the examples used in this module's documentation)

1. fetch changes to a collection of documents across a number of feed providers
2. render +upload PDFs for each of the documents that have changed.
3. notify consumers of a change to the respective PDF (e.g., send an e-mail).

I can see a number of ways that one could report on the status of where a workflow is at in processing; however, I would appreciate some second and third opinions on how to approach reporting on the state of a workflow that leverages SlmQueue. Pros/Cons are also appreciated (e.g., dealing with data collisions etc.).

Possibility 1: --> each job inserts or updates an array into an object cache like redis and assembled into something like:

{
    "job_id": 1443335458,
    "job_type": "pdf_bulk_update",
    "job_status": "in-progress",
    "components": [
        {
            "type": "feed_provider",
            "feed_provider_id": 1,
            "status": "searching"
        }
    ]
}

1. Each worker inserts records into a database or object cache and a status method aggregates the results and calculates status.
2. Some other possibility that I am not yet considering.

Any and all thoughts are appreciated here! I'm assuming that this is a question that just about any developer has to deal with when it comes to pushing tasks to background processing when dealing with client-facing applications.

Thanks!

[jackdpeterson]

Hello all,

For those who have this issue, the following ZF2 module is designed to solve this problem as a whole.

I anticipate marking it as stable after some additional testing in a real environment.

https://packagist.org/packages/jackdpeterson/immutable-state-status-tracker

[di-sysops]

Thanks!