uclahs-cds / tool-NFTest

CLI to automate Nextflow pipeline testing
GNU General Public License v2.0
11 stars 1 forks source link

Add `--report` flag to save JSON test summary #71

Closed nwiltsie closed 2 months ago

nwiltsie commented 2 months ago

Description

This adds a --report flag that writes out a JSON test summary file alongside the NFTest log file. As an example, see /hot/software/pipeline/pipeline-convert-BAM2FASTQ/Nextflow/development/unreleased/main/log-nftest-20240909T225839Z.json:

{
  "start": "2024-09-09T22:58:39.724080+00:00",
  "passed_tests": {
    "test_bam2fastq": 112.80969
  },
  "skipped_tests": {},
  "errored_tests": {},
  "failed_tests": {},
  "cpus": 2,
  "end": "2024-09-09T23:00:32.534372+00:00",
  "success": true
}

This is related to but not quite the same thing as #7 - this saves the summary to a file for easier downstream parsing without printing anything new to the console.


I implemented this with a new NFTestReport class. I tried to keep it as modular and distinct as possible from the existing logic, so the main structural changes are:

Along the way I found a hidden bug where common.resolve_single_path() raised ValueErrors, which were not being caught by NFTestRunner.main(). That meant that NFTest would crash on the first NFTestAssert with 0 or 2+ matching files. I fixed that by:

Checklist

[^1]: UCLA Health reaches $7.5m settlement over 2015 breach of 4.5m patient records [^2]: The average healthcare data breach costs $2.2 million, despite the majority of breaches releasing fewer than 500 records. [^3]: Genetic information is considered PHI. Forensic assays can identify patients with as few as 21 SNPs [^4]: RNA-Seq, DNA methylation, microbiome, or other molecular data can be used to predict genotypes (PHI) and reveal a patient's identity.

  To automatically exclude such files using a .gitignore file, see here for example.

yashpatel6 commented 2 months ago

I'll get to this tomorrow!