search

nextstrain / .github

8 stars 11 forks source link

pathogen-repo-build: Generate a Snakemake report for workflow #100

Open joverlee521 opened 1 month ago

joverlee521 commented 1 month ago

Context

Originally proposed in https://github.com/nextstrain/cli/issues/373#issuecomment-2174603786

It'd be helpful to have a summary of workflow stats in the form of a Snakemake report for our automated pathogen workflows.

joverlee521 commented 1 month ago

At first I thought this would be pretty straight-forward:

  1. Add another step to the GH Action workflow to run nextstrain build . --report
  2. Add report.html to the list of paths for the build artifact

However, the command nextstrain build . --report will not always work since the run input allows any arbitrary shell command, e.g. if the run command is nextstrain build ingest ..., we would need to run nextstrain build ingest --report.

joverlee521 commented 1 month ago

For non-aws-batch runtimes, I think this can be achieved directly through the run input

run: nextstrain build ingest; nextstrain build ingest --report
joverlee521 commented 1 month ago

For non-aws-batch runtimes, I think this can be achieved directly through the run input

run: nextstrain build ingest; nextstrain build ingest --report

Yup, this totally worked in zika/ingest (wip, artifact)

joverlee521 commented 1 month ago

We cannot follow the same pattern in the aws-batch runtime because of the wait for the AWS Batch job to finish...

Ah, we can download the Snakemake files into the current dir then generate the report (dependent on release with https://github.com/nextstrain/cli/pull/374):

nextstrain build --aws-batch --attach <job-id> --download ".snakemake/" . ;
nextstrain build . --report

I think this does not have to be built into the pathogen-repo-build workflow as it can just be run as a separate job in the caller workflow (small block on https://github.com/nextstrain/.github/issues/79)

I guess we can see how helpful Snakemake reports are for now with these alternative methods of generating them before we add more complexity to the pathogen-repo-build workflow.

tsibley commented 1 month ago

(dependent on release with nextstrain/cli#374):

@joverlee521 Want to release 8.5.0 of Nextstrain CLI?

joverlee521 commented 1 month ago

(dependent on release with nextstrain/cli#374):

@joverlee521 Want to release 8.5.0 of Nextstrain CLI?

Yup! Will do ~now.