[ENHC0010012A] Workflow executions: Cleanup job + service code flow

[JeffreyThiessen]

What does this PR do and why?

Describe in detail what your merge request does and why. This PR modifies the code flow of jobs and services to have a final step where a "Cleanup" step is performed after a Workflow Execution has reached one of it's 3 end states of completed, canceled, or error.

This PR does not contain the changes that do the cleanup. That will be handled in a separate PR. In this PR, the cleanup job is queued, and when run, runs the service which simply marks cleaned as true. This is so we can test all cases that pertain to the job/service code flow.

Main change that is visible to UI is that workflow executions are not deletable until after the cleanup service has run. So there is a 30 second job queue delay between a run being completed/canceled/error and being deletable.

Fixes #521 Dependant on #605

This service will clean up files generated though a Workflow Executions life that are no longer needed. The job handles scheduling the cleanup service.

The simple explanation of the code flow is

The existing code flow for Workflow Executions is: initial >> [ submitted, running, etc] >> One of end state [completed, canceled, error] >> WE is deletable

New code flow is: initial >> [ submitted, running, etc] >> One of end state [completed, canceled, error] >> Cleanup job >> .cleaned? >> WE is deletable

The full code flow diagram is

This diagram is also included on a new docs page.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

NOTE!!!!! To test this branch, you need to have the updated Sapporo branch and use the new commands to start it. This will allow the run outputs to be deleted correctly by irida next. Use the irida-next branch of phac-nml/sapporo-service and use the new commands found here https://github.com/phac-nml/sapporo-service/blob/irida-next/README.irida-next.md

Run a workflow execution and verify that after completion/cancelation/error state, the run is not immediately deletable. Then after the cleanup job/service has run, the input/output files for the workflow should be cleaned up, and the workflow execution should be deletable via the web ui.

PR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• [x] I have evaluated the PR acceptance checklist for this PR.

[JeffreyThiessen]

ENHC0010012

[github-actions[bot]]

Simplecov Report

Covered	Threshold
92.35%	90%