[Working version for Small 2d and BRT] Small 2D and BRT fixes

[annshress]

Changes

• Add notify api running calls as soon as the flow runs start
• czi flow run uses async/await. Wait for prefect future results before sending it to async calls
• Remove subflow for small dm conversion flow
• Revert Completion/Failed to original success/error api calls on notify-api-completion
• Fix BRT flow: Add dependencies between task runs (replaces wait_for upstream tasks)... Increase cluster size for BRT workflow to be able to handle large number of inputs.
• cleanup workdir moved to hooks from tasks.
• Add error for task as messages in callback response using task failure hooks (uses file i/o)

Shortcomings

• Using file i/o to store errors (this is used to collect error messages to collate in callback response; also, here to store which temporary file locations need to be cleared by on_completion/on_failure hooks) --- Two functions of interest in utils.py are: get_flow_run_tmpdirs and store_exception_hook (for two respective issues).

This PR doesn't introduce any:

• [x] Binary files
• [x] Temporary files, auto-generated files
• [x] Secret keys
• [x] Local debugging print statements
• [x] Unwanted comments (e.g: # Gets user from environment for code os.environ['user'] )

This PR contains valid:

• [ ] tests

[github-actions[bot]]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
em_workflows
file_path.py					222
em_workflows/dm_conversion
flow.py
em_workflows/lrg_2d_rgb
flow.py
em_workflows/utils
utils.py					72, 81-84, 95-108, 406, 518, 595-598, 607-620
Project Total

This report was generated by python-coverage-comment-action

[annshress]

Closing...

• file i/o is not the most optimal approach on hooks [ solution: get task run result from hooks directly rather than relying on a hack-y approach ]