Queue communication

[guidj]

Neumann should read jobs from queue, and send responses to a response queue for the OMS:

{
    "status": Http Status Codes,
    "message": "str",
    "task": {
        "id": "str",
        "state": "WAITING/RUNNING/CANCELLED/FAILED"
    }
}

[guidj]

There will be a main thread, that will read jobs from a queue. This thread will create an entry into a K-V store, potentially Redis or ZODB.

• Once the job is actually submitted to the work queue, it's status will be set to PENDING.
• When the job starts execution, it's status will be set to RUNNING
• Once complete, or failed, the status will be set to DONE or FAILED accordingly

The outcome or result of the job/task will then we submitted to a result queue for the OMS, and kept for a certain period of time, or until it is acknowledged.

In order for workers to update the status of a job, there should be a queue that will be consumed by the main thread as well. This will be prevent concurrency problems in the data store.

If the Restful endpoint will be made available, then it would need to communicate with the main thread that submit jobs.

Queues (internal? and external):

• Job submission
• Job status update
• Job cancellation

[jocki]

To make neumann more simple, I think it is better if neumann doesn't manage the status of its jobs. neumann will just emit events (by sending them as event messages to oms) such as RUNNING and DONE. Then let OMS stores these events and performs actions such as re-sending command message if it received FAILED event message.

These events can be send from every individual Luigi's tasks or we can tap into Luigi event notification itself for global solution.

[st3wart]

Can do.. On 20 Oct 2015 12:29, "jocki" notifications@github.com wrote:

To make neumann more simple, I think it is better if neumann doesn't manage the status of its jobs. neumann will just emit events (by sending them as event messages to oms) such as RUNNING and DONE. Then let OMS stores these events and performs actions such as re-sending command message if it received FAILED event message.

These events can be send from every individual Luigi's tasks or we can tap into Luigi event notification itself for global solution.

— Reply to this email directly or view it on GitHub https://github.com/predictry/neumann/issues/121#issuecomment-149428799.

[jocki]

Added code to emit the following Luigi's event for every top-level task execution: START, SUCCESS, FAILURE and PROCESSING_TIME.

Top-level task are all tasks that inherit from EventEmitter. In current code, they're TaskImportRecordIntoNeo4j, TaskRunRecommendationWorkflow, and TaskRunTrimDataWorkflow.