search

broadinstitute / cromwell-tools

A collection of Python clients and accessory scripts for interacting with the Cromwell
https://cromwell-tools.readthedocs.io
BSD 3-Clause "New" or "Revised" License
22 stars 9 forks source link

Add utilities to inject & collect monitoring log info #37

Open danxmoran opened 6 years ago

danxmoran commented 6 years ago

Cromwell-on-GCP exposes a runtime option to inject a monitoring script into every task in a workflow, but (as far as I can tell) doesn't expose a way to collect the generated log paths via its API. For users who want to make use of monitoring, I think adding a new collect-monitoring-logs command to cromwell-tools would be useful. The command would:

  1. Take a workflow ID,
  2. Scrape the corresponding Cromwell execution directory for files named monitoring.log,
  3. Output a dict of task ID -> raw monitoring log output.

Users could then bolt their own interpreters onto this command to parse their specific log outputs.

@benjamincarlin has been working on code to do this, and we thought it'd be good to contribute back here instead of setting up yet-another-repo. Would a PR be welcome?

kbergin commented 6 years ago

Sounds useful! @rexwangcc @samanehsan any thoughts here? A PR is an appropriate way to contribute and we're happy to review. We 👍 anything that doesn't produce another repository!!! Thanks

samanehsan commented 6 years ago

Yes, that sounds great!

rexwangcc commented 6 years ago

That sounds great! By the way, I opened a WIP PR: https://github.com/broadinstitute/cromwell-tools/pull/39 which shows @ambrosejcarr previous efforts and parts of our agreements in our other services (e.g. Google docstring, pytest for unittests...) But feel free to make the PR against the current master, I can rebase on that later!