Ensure data for negative controls is delivered to customers

[karlnyr]

Description

Negative controls require all reads linked to their housekeeper bundle and have the read counts added to the sample in statusdb and LIMS.

Suggested solution

When we parse the demultiplexing stats we do get the number of reads from the sample, the next step would be to (a) check if a sample is a negative control or (b) find the metagenomic negative control on a flow cell, and handle it differently.

This can be closed when

Negative controls have their reads added to the sample bundle, statusdb, and LIMS disregarding read quality.

Clarification

Negative controls by definition won't pass our QC after sequencing and won't have the fastq files linked to them. Negative controls should always have their data transferred. These can be used by the customer to filter out any noise from the prep. Also, in the new metagenomic analyses they are needed.

1. Negative controls do not pass sequencing QC
2. Negative controls fastq are not stored in their respective housekeeper sample bundle
3. Negative control samples are not added or updated in status db in the IlluminaSampleSequencingMetrics table.
4. Negative control samples need to be added to LIMS.

The above four steps are required to pass.

Second Clarification

• Customers want ALL the data produced, and we are not providing the data generated for negative controls
• The new taxprofiler workflow needs the data from the negative control to run

[karlnyr]

Discussion

• Can we use bad-quality reads
• Add a property to the sample model which is "sample_is_metagenomic"-sh
• Should we do this for all negative controls, and deal with the repercussions later?

Decision

• [ ] Implement this for all negative controls
  • [x] If a sample is negative control then add any reads at all independent of quality
• [ ] Check that pipelines where this could affect (microsalt, mutant) do not impact it negatively
  • [x] Create a ProdBioInfo task to start a microsalt case including reads with poor quality. @karlnyr

[karlnyr]

Plan is to analyse eternaltahr with its negative control ACC9713A25 which has reads and of low quality, requesting flow cell HVNHGDSX2 to fetch the data back to hasta so that we can add it to ACC9713A25 temporarily.

[beatrizsavinhas]

• [X] Demultiplexed flowcell
• [X] Created housekeeper bundle and added fastq files for negative control sample ACC9713A25
• [X] Retrieved archived files for all samples
• [x] Decompressed all files
• [x] Started microsalt analysis for case eternaltahr

[karlnyr]

Analysis completed without any issues. This means that we can now begin the process of adding data for all negative controls to hk as well! @Clinical-Genomics/sysdev, could you prioritize this? The current situation is very prone to mistakes, and the metagenomic samples are quite time-sensitive.

[diitaz93]

Technical refinement

Questions

• Does this concern only MIcroSALT or other pipelines? @karlnyr

[karlnyr]

It is not pipeline relevant. This is on an organisation level where we would like to store all reads for samples that are negative controls @diitaz93

[Vince-janv]

@karlnyr What issue would this resolve? Is it a requirement for taxprofiler being implemented in production?

[karlnyr]

This would resolve two things:

• Automatic transfer of fastq files for negative controls
• Enable taxprofiler to use negative control reads in the future

[karlnyr]

Edited issue, as this would be changed for all negative controls, not specifically for metagenomic ones.

[Vince-janv]

Technical Refinement

• Use sample.control to determine if low-quality fastq files should be added to housekeeper (in post-processing flow)
• Also update the sample.reads for negative controls

[beatrizsavinhas]

Changes reverted.

Blocked until the changes can be implemented in LIMS.