chanzuckerberg / idseq-workflows

Portable WDL workflows for IDseq production pipelines
https://idseq.net/
MIT License
31 stars 12 forks source link

Migrate status JSON files out of idseq-dag #80

Closed mlin closed 3 years ago

mlin commented 3 years ago

Delete the code for creating these JSON files (to be read by webapp to display pipeline status) from idseq-dag, in favor of posting them from a miniwdl plugin.

The status JSON includes a markdown description for each step, which in some cases idseq-dag customized dynamically to reflect details of the specific inputs. We keep that idseq-dag logic and transmit the description through a WDL String output from each task to be read by the plugin.

mlin commented 3 years ago

Good point @morsecodist -- per @tfrcarvalho suggestion, I restored the existing code to update _status.json within idseq-dag, and changed the plugin diff to create _status2.json instead. This will make a smoother transition, we can re-delete the idseq-dag bits once the webapp is confirmed working with status2.json. (The diff still deletes some invocations of update_status_json_file() that are in dead code anyway.)

kislyuk commented 3 years ago

Just noting down a follow-on thought that we've discussed a few times: it would be great to minimize the number of tasks (pipeline steps) that have dynamic descriptions, and for those that have static descriptions, write those down as markdown comments in the WDL file itself, so the frontend can consume them from the static copy. (Building on that, for dynamic descriptions, we could use a templating convention where the docstring uses a jinja or ERB template with values that can be filled in from task inputs or outputs.)

mlin commented 3 years ago

They certainly were an inconvenience! One limitation is that WDL doesn't yet have multi-line string literals (task commands excepted), so it's inconvenient to put a lengthy markdown in them. That's been discussed in the standards forum, but hasn't made its way through yet.