seallard
commented
9 months ago Digging through logs for the most recent missing case:
...
2024-01-14 03:44:06 hasta.scilifelab.se cg.meta.workflow.analysis[4367] INFO Linking: /home/proj/production/housekeeper-bundles/ACC13659A28/2024-01-13/22FYWJLT3_ACC13659A28_S340_L008_R2_001.fastq.gz -> /home/proj/production/microbial/fastq/trustedmidge/
ACC13659A28/ACC13659A28_22FYWJLT3_L8_2.fastq.gz
2024-01-14 03:44:15 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] INFO Saved config /home/proj/production/microbial/queries/trustedmidge.json
2024-01-14 03:44:15 hasta.scilifelab.se cg.meta.workflow.microsalt.microsalt[4367] ERROR No result directory found for trustedmidge with project id ACC13659
2024-01-14 03:44:15 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] WARNING Trailblazer warning: Could not track analysis progress for case trustedmidge! MissingAnalysisDir
2024-01-14 03:44:15 hasta.scilifelab.se cg.meta.workflow.analysis[4367] INFO Action running set for case trustedmidge
2024-01-14 03:44:15 hasta.scilifelab.se cg.utils.commands[4367] INFO Running command /home/proj/production/bin/miniconda3/bin/conda run --name P_microSALT /home/proj/production/bin/miniconda3/envs/P_microSALT/bin/microSALT analyse /home/proj/production/
microbial/queries/trustedmidge.json --input /home/proj/production/microbial/fastq/trustedmidge
2024-01-14 03:44:39 hasta.scilifelab.se cg.utils.commands[4367] CRITICAL Call ['/home/proj/production/bin/miniconda3/bin/conda run --name P_microSALT', '/home/proj/production/bin/miniconda3/envs/P_microSALT/bin/microSALT', 'analyse', '/home/proj/product
ion/microbial/queries/trustedmidge.json', '--input', '/home/proj/production/microbial/fastq/trustedmidge'] exit with a non zero exit code
2024-01-14 03:44:39 hasta.scilifelab.se cg.utils.commands[4367] CRITICAL Error: stepping, database is locked (5)
ERROR - Database integrity failed! Lock-state detected!
ERROR conda.cli.main_run:execute(124): `conda run /home/proj/production/bin/miniconda3/envs/P_microSALT/bin/microSALT analyse /home/proj/production/microbial/queries/trustedmidge.json --input /home/proj/production/microbial/fastq/trustedmidge` failed. (
See above for error)
2024-01-14 03:44:39 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] ERROR Failed to run analysis!
2024-01-14 03:44:39 hasta.scilifelab.se cg.meta.workflow.analysis[4367] INFO Action None set for case trustedmidge
2024-01-14 03:44:39 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] ERROR Unspecified error occurred: Command '['/home/proj/production/bin/miniconda3/bin/conda run --name P_microSALT', '/home/proj/production/bin/miniconda3/envs/P_microSALT/bin/
microSALT', 'analyse', '/home/proj/production/microbial/queries/trustedmidge.json', '--input', '/home/proj/production/microbial/fastq/trustedmidge']' returned non-zero exit status 255.
Aborted!Two issues: missing result dir and db lock.
Another missing case
2024-01-14 03:35:58 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] INFO Starting Microsalt workflow for wiredbarnacle
... linking files
2024-01-14 03:36:33 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] INFO Saved config /home/proj/production/microbial/queries/wiredbarnacle.json
2024-01-14 03:36:33 hasta.scilifelab.se cg.meta.workflow.microsalt.microsalt[4367] ERROR No result directory found for wiredbarnacle with project id ACC13652
2024-01-14 03:36:33 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] WARNING Trailblazer warning: Could not track analysis progress for case wiredbarnacle! MissingAnalysisDir
2024-01-14 03:36:33 hasta.scilifelab.se cg.meta.workflow.analysis[4367] INFO Action running set for case wiredbarnacleThe oldest missing case
2024-01-14 02:27:38 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] INFO Starting Microsalt workflow for smashingwhippet
... linking files
2024-01-14 02:28:55 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] INFO Saved config /home/proj/production/microbial/queries/smashingwhippet.json
2024-01-14 02:28:55 hasta.scilifelab.se cg.meta.workflow.microsalt.microsalt[4367] ERROR No result directory found for smashingwhippet with project id ACC13525
2024-01-14 02:28:55 hasta.scilifelab.se cg.cli.workflow.microsalt.base[4367] WARNING Trailblazer warning: Could not track analysis progress for case smashingwhippet! MissingAnalysisDir
2024-01-14 02:28:55 hasta.scilifelab.se cg.meta.workflow.analysis[4367] INFO Action running set for case smashingwhippet
2024-01-14 02:28:55 hasta.scilifelab.se cg.utils.commands[4367] INFO Running command /home/proj/production/bin/miniconda3/bin/conda run --name P_microSALT /home/proj/production/bin/miniconda3/envs/P_microSALT/bin/microSALT analyse /home/proj/production/microbial/queries/smashingwhippet.json --input /home/proj/production/microbial/fastq/smashingwhippet
Some microsalt cases are not showing in tb. The oldest missing case is from 2023-12-18. The most recent missing case is from 2024-01-09. There are microsalt analyses in trailblazer after, during and before this period. So it seems to be a bug only affecting certain microsalt cases.
Missing cases:
Proposed solution
Change the result directory for microsalt cases to just be the case name instead of lims project id + random timestamp.
The current naming causes issues since the result path needs to be passed when adding a pending analysis to trailblazer. The timestamp part is random, which makes it impossible to provide before the out directory has been created by the pipeline.