This adds a --report flag that writes out a JSON test summary file alongside the NFTest log file. As an example, see /hot/software/pipeline/pipeline-convert-BAM2FASTQ/Nextflow/development/unreleased/main/log-nftest-20240909T225839Z.json:
This is related to but not quite the same thing as #7 - this saves the summary to a file for easier downstream parsing without printing anything new to the console.
I implemented this with a new NFTestReport class. I tried to keep it as modular and distinct as possible from the existing logic, so the main structural changes are:
There is a new TestResult enumeration defined in common.py.
NFTestCases now have a status attribute that is initialized to TestResult.PENDING. NFTestCase.test() sets that to PASSED, FAILED, ERRORED, or SKIPPED as appropriate.
NFTestRunner.main() creates an NFTestReport instance, and uses report.track_case(case) as a context manager around each test case. If --report was passed it writes out that report after all the tests have completed.
Along the way I found a hidden bug where common.resolve_single_path() raised ValueErrors, which were not being caught by NFTestRunner.main(). That meant that NFTest would crash on the first NFTestAssert with 0 or 2+ matching files. I fixed that by:
Moving resolve_single_path into NFTestAssert
Updating NFTestAssert to throw custom exception classes derived from NFTestAssertionError, rather than ValueError and AssertionError.
Updating NFTestRunner.main() to catch NFTestAssertionError rather than AssertionError.
Checklist
[x] This PR does NOT contain Protected Health Information (PHI). A repo may need to be deleted if such data is uploaded. Disclosing PHI is a major problem[^1] - Even a small leak can be costly[^2].
[x] This PR does NOT contain germline genetic data[^3], RNA-Seq, DNA methylation, microbiome or other molecular data[^4].
[x] This PR does NOT contain other non-plain text files, such as: compressed files, images (e.g..png, .jpeg), .pdf, .RData, .xlsx, .doc, .ppt, or other output files.
To automatically exclude such files using a .gitignore file, see here for example.
[x] I have set up or verified the main branch protection rule following the github standards before opening this pull request.
[x] The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
[x] I have added the major changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.
Description
This adds a
--report
flag that writes out a JSON test summary file alongside the NFTest log file. As an example, see/hot/software/pipeline/pipeline-convert-BAM2FASTQ/Nextflow/development/unreleased/main/log-nftest-20240909T225839Z.json
:This is related to but not quite the same thing as #7 - this saves the summary to a file for easier downstream parsing without printing anything new to the console.
I implemented this with a new
NFTestReport
class. I tried to keep it as modular and distinct as possible from the existing logic, so the main structural changes are:TestResult
enumeration defined incommon.py
.NFTestCase
s now have astatus
attribute that is initialized toTestResult.PENDING
.NFTestCase.test()
sets that toPASSED
,FAILED
,ERRORED
, orSKIPPED
as appropriate.NFTestRunner.main()
creates anNFTestReport
instance, and usesreport.track_case(case)
as a context manager around each test case. If--report
was passed it writes out that report after all the tests have completed.Along the way I found a hidden bug where
common.resolve_single_path()
raisedValueError
s, which were not being caught byNFTestRunner.main()
. That meant that NFTest would crash on the firstNFTestAssert
with 0 or 2+ matching files. I fixed that by:resolve_single_path
intoNFTestAssert
NFTestAssert
to throw custom exception classes derived fromNFTestAssertionError
, rather thanValueError
andAssertionError
.NFTestRunner.main()
to catchNFTestAssertionError
rather thanAssertionError
.Checklist
[x] This PR does NOT contain Protected Health Information (PHI). A repo may need to be deleted if such data is uploaded.
Disclosing PHI is a major problem[^1] - Even a small leak can be costly[^2].
[x] This PR does NOT contain germline genetic data[^3], RNA-Seq, DNA methylation, microbiome or other molecular data[^4].
[^1]: UCLA Health reaches $7.5m settlement over 2015 breach of 4.5m patient records [^2]: The average healthcare data breach costs $2.2 million, despite the majority of breaches releasing fewer than 500 records. [^3]: Genetic information is considered PHI. Forensic assays can identify patients with as few as 21 SNPs [^4]: RNA-Seq, DNA methylation, microbiome, or other molecular data can be used to predict genotypes (PHI) and reveal a patient's identity.
.png
, .jpeg
),.pdf
,.RData
,.xlsx
,.doc
,.ppt
, or other output files.To automatically exclude such files using a .gitignore file, see here for example.
[x] I have read the code review guidelines and the code review best practice on GitHub check-list.
[x] I have set up or verified the
main
branch protection rule following the github standards before opening this pull request.[x] The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
[x] I have added the major changes included in this pull request to the
CHANGELOG.md
under the next release version or unreleased, and updated the date.