Incorrect status of flowcell and now fastq files are linked twice in HK and read count in cgstats is wrong

[moahaegglund]

Flowcell HV7VTCCXY had status removed even though it in reality was on disc (present in the demultiplexed folder but not in HK) from being demultiplexed in 2019. The structure in the demultiplexed folder and the naming of the files has changed leading to the fastq files being included twice in Housekeeper.

$ housekeeper get file ACC5113A10
2021-04-15 11:06:57 hasta.scilifelab.se housekeeper.cli.core[246652] INFO Use database mysql+pymysql://housekeeper:X1yek09lW14jG7XU@localhost:3308/housekeeper
2021-04-15 11:06:57 hasta.scilifelab.se housekeeper.cli.core[246652] INFO Use root path /home/proj/production/housekeeper-bundles
2021-04-15 11:06:58 hasta.scilifelab.se housekeeper.store.api.find[246652] INFO Fetching files from bundle ACC5113A10
                                 📜 Files table 📜
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ ID     ┃ File name                                            ┃ Tags             ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ 850728 │ HV7VTCCXY-l2t21_640692_AGATCGCA_L002_R2_001.fastq.gz │ fastq, HV7VTCCXY │
│ 850729 │ HV7VTCCXY-l2t11_640692_AGATCGCA_L002_R1_001.fastq.gz │ fastq, HV7VTCCXY │
│ 850730 │ HV7VTCCXY-l2t21_640692_AGATCGCA_L002_R1_001.fastq.gz │ fastq, HV7VTCCXY │
│ 850731 │ HV7VTCCXY-l2t21_640692_S2_L002_R2_001.fastq.gz       │ fastq, HV7VTCCXY │
│ 850732 │ HV7VTCCXY-l2t11_640692_AGATCGCA_L002_R2_001.fastq.gz │ fastq, HV7VTCCXY │
│ 850733 │ HV7VTCCXY-l2t21_640692_S2_L002_R1_001.fastq.gz       │ fastq, HV7VTCCXY │
│ 850734 │ HV7VTCCXY-l2t11_640692_S2_L002_R1_001.fastq.gz       │ fastq, HV7VTCCXY │
│ 850735 │ HV7VTCCXY-l2t11_640692_S2_L002_R2_001.fastq.gz       │ fastq, HV7VTCCXY │
└────────┴──────────────────────────────────────────────────────┴──────────────────┘

The renaming of files during linking to MIP prevent the files from being included twice in that analysis. (For information I've solve this manually for sample ACC5113A11 while linking undetermined fastq files as I had to start a prio case.)

This has also affected cgstats, samples have the wrong amount of reads:

$ cgstats sample ACC5113A10
1713518192

The sample has 857M reads according to LIMS.

How can we fix this is cgstats and HK? How can we prevent this from happening again?

[barrystokman]

Remove the flowcell (and samples) from cgstats and manually add it again. Remove the files you don't want MIP to use from HK.

As for the incorrect flowcell status:

1. short term: check hasta before requesting a flowcell from PDC (action: team production)
2. longer term: find all incorrect flowcell statuses and fix them (action: Team Trocadero)

[emmser]

How can we prevent that the files are being added twice to HK?

[barrystokman]

We need to make sure we don't add them a second time, so before fetching the flowcell from PDC and demuxing again we need to check if the flowcell is actually ondisk. Unfortunately, the flowcell status in statusdb is not always correct. You need to check manually (for now) if the flowcell exists on disk. If that is the case, we don't need to fetch and demux. This is something the production team can do.

Of course doing things manually is not how we want to do things, so we need to check and if needed fix all flowcell statuses. This is something one of the developers will do.

[moahaegglund]

When fetching from PDC works, doesn't it start automatically if MIP tries to start a case where the flowcell of 1 of 3 samples has status removed?

[barrystokman]

That's a good point and I think you are correct.

[moahaegglund]

Yes, I found it here: function all_flowcells_on_disk in https://github.com/Clinical-Genomics/cg/blob/d3c802794ec3d6eb6c3fb327659c9f306dceb0d4/cg/meta/workflow/analysis.py.

[moahaegglund]

The files will not be linked twice to the MIP analysis but in case of a fastq delivery it will be wrong (happened before). I don't know how the other pipelines would handle this.

[emmser]

And also in some cases I've seen that the focus on disk are only partly kept, some Fastq-filrs have been removed while some are still there, in those cases we still want to be able to demux the whole flowcell without adding files twice. Is it impossible to do this check automatically?

[emmser]

Remove the files you don't want MIP to use from HK.

To handle the files manually in this way is not a safe way to do it, it's very easy to do it wrong

[moahaegglund]

I think we should wait to do anything until the prio case fastfalcon is delivered, the index sample is from this flowcell. But would it be possible to delete everything in HK connected to this flowcell, remove it from hasta (/home/proj/production/demultiplexed-runs/), cgstats and redo the demultiplexing? Would that solve this in short term without too much manual work? Or maybe its OK to remove the flowcell from disc? (I don't know if that is possible? I remember we all being confused about the removal of flowcells before summer.)

[barrystokman]

@moahaegglund that works