Open discussion on job states/jobflow integration

[davidwaroquiers]

In light of future developments/integration with/in jobflow, I open this issue to discuss the possible states of a job.

The current states of a job are:

• WAITING = "WAITING" : This job is waiting for another (or several other) job(s) to complete (or possibly fail, if something similar to "allow_fizzled_parents" in fireworks is implemented) before being able to start (and becoming in a READY state).
• READY = "READY" : This job can start. It has no dependencies on previous jobs or all previous dependent jobs have completed or failed (see above for allow_fizzled_parents note).
• ONGOING = "ONGOING" : This job is "running" or "ongoing". This means something is being done for this job. In the context of jobflow-remote, this means the job can be in one of the remote states (e.g. being checked out, uploaded to the worker, submitted, actually running, ...
• REMOTE_ERROR = "REMOTE_ERROR" Something went wrong with the remote execution (e.g. the files could not be uploaded to the worker, or downloaded from it, e.g. if it is not accessible, or if there is no space left on the worker or on the local machine where the files have to be downloaded, ...).
• COMPLETED = "COMPLETED" : The job has successfully completed.
• FAILED = "FAILED" : The job has failed. This is usually due to an explicit error raised by the job due to something that went wrong (could be e.g. the simulation did not converge, ... this is something that is/should be implemented in the jobs/flows by the developer). This could also be because of a bug/error (e.g. a python error is raised because some file does not exist), of course in production when flows have been full proofed it should not happen very often.
• PAUSED = "PAUSED" : The job has been set in pause by the user. This is currently not implemented (at least, there is no cli command allowing to do that even though the PAUSED state should already be handled if it is set manually in the database). The idea would be to allow to "unpause" a paused job of course.

I can think of two additional states that could prove useful:

• CANCELLED = "CANCELLED" : The job has been cancelled (manually) by the user.
• STOPPED = "STOPPED : The job has been stopped automatically by jobflow (e.g. from a Response with stop_children or stop_jobflow).

Both cases would probably be handled exactly the same way as PAUSED but would/could be distinct just for the sake of a user's perspective. (although a cancelled job is not meant to be "uncancelled" and a stopped job is not meant to be "unstopped")

It is not yet clear whether jobflow-remote will be integrated completely or partly inside jobflow or left as a "third-party" (possibly mandatory or optional deps of jobflow). I would think that in any case it might be good to use the same JobState in the new queue. One point about the REMOTE_ERROR is that we chose the name instead of just "ERROR" so that there is no confusion between ERROR and FAILED. (if job states go in jobflow, REMOTE_ERROR would not have a "general" sense, we "might" think of changing the name but I am really not sure, and if so, to what ? EXEC_ERROR ? something else ?)

@gpetretto Any other comments on this ? @utf What do you think ? Might be good to have a meeting to discuss this. Do you think we could/should include someone else in the discussion ? I will send you an email to see when we can discuss this and the new queue.

[davidwaroquiers]

One thing which could be nice is to have names that all have distinct first letter so that a single letter would also correspond to a single state (currently READY is R and REMOTE_ERROR is RE). We could have:

• WAITING = "WAITING"
• READY = "READY"
• ONGOING = "ONGOING"
• EXEC_ERROR = "EXEC_ERROR" # instead of REMOTE_ERROR
• DONE = "DONE" # instead of COMPLETED
• FAILED = "FAILED"
• PAUSED = "PAUSED"
• STOPPED = "STOPPED"
• CANCELLED = "CANCELLED"

Not sure if it is that important though.