Parameter file support

[sylvinite]

Description

The features of this PR primarily concerns end-users/bioinformaticians/internals

This is a summary of the changes made.

• [x] Added sampleinfo file parser
• [x] Condense project/sample/collection into "supercollection"
• [x] Sample info file padded with default info
• [x] Add sample info generator
• [x] Sample info file automatically verified in CLI
• [x] Added database testing
• [x] ST tracking system now more verbose
• [X] Integrated rerun flag into all finish commands

Miniscule:

• [x] Update readme
• [x] Replace scraper dict calls with .get()
• [x] Debug level log statements moved to debug
• [x] CLI input is passed through classes with struct run_settings
• [x] Renamed param to sample_info
• [x] sys.exit replaced with click.abort where relevant
• [x] Instances renamed to more informative names
• [X] One letter variables chosen more informatively
• [X] Comments replaced by more informative method names
• [x] .get method applied to fetching dict values to remove NoneType
• [x] Change CLI tests to display more info on crash, and look for proper end of function.

Primary function of PR

• [ ] Hotfix
• [ ] Patch
• [ ] Minor functionality improvement
• [ ] New type of analysis
• [x] Backward-breaking functionality improvement
• [x] This change requires internal documents to be updated
• [ ] This change requires another repository to be updated

Testing

This is a description of the tests necessary to verify the stability of the PR. In order to verify the stability of the PR, the following must be tested:

• The code that parses a sample info file

• The code that validates the integrity of the sample info file (rejects illegal sample info files within reason)

• The microSALT analysisstill works for a project, sample and collection (mixed samples)

• The code responsible for parsing the sampleinfo file works with correct values

• _bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-microsalt-stage.sh param_file_

• us

• _source activate S_microSALT_

• cd microSALT/tests/testdata

• microSALT analyse MIC3109.json --input /home/proj/stage/microbial/fastq/MIC3109/

• microSALT analyse MIC4107.json --input /home/proj/stage/microbial/fastq/MIC4107/

• microSALT analyse MIC4109.json --input /home/proj/stage/microbial/fastq/MIC4109/

• microSALT analyse ACC5551.json --input /home/proj/stage/microbial/fastq/ACC5551/

_Verify that the results for projects MIC3109, MIC4107, MIC4109 & ACC5551 are consistent with the results attached to AMSystem doc 1490, MicrobialWGS.xlsx

Extended routine:

• Reduce MIC3109 to a single sample and make sure microSALT still functions
• microSALT analyse MIC3109A2.json --input /home/proj/stage/microbial/fastq/MIC3109/MIC3109A2
• Try to break the sample json file as much as possible, and verify that microSALT either halts, resolves or crashes gracefully

Test results

The values of MIC3109, MIC4107, MIC4109 & ACC5551 are consistent with the results attached to AMSystem doc 1490, Microbial_WGS.xlsx with one exception. The exception is that the provided json files have been scrubbed of Sample ID, thus incorrectly flagging NTC samples as failing.

Running a json file only containing info for MIC3109A2 successfully generated a Typing and QC report with correct results.

Modifying the inbound json in order to make it unusable consistently generated the error ERROR - Parameter XXX needs to be provided in sample json. Formatting example:

Sign-offs

• [x] Code tested by @sylvinite
• [x] Approved to run at Clinical-Genomics by @patrikgrenfeldt