TapeStation data from csv file

annaengstrom commented 4 years ago

Lims Issue

Work Flow: Twist v2 Protocol: Library validation QC Twist v1 Step: TapeStation QC Twist

Description: The Tapestation instrument provides a csv file

Lims should extract Limd IDs from file header: Sample Description Lims should extract Average size from file header: Average size

Script should add fill in the values. It is possible that not all lims ids in the step are present in csv file. It is possible that the lims ID's are not in the correct order. It is the Lims ID's that should be matched with each other and average size should be filled in.

The same procedure should be added to all TapeStation QC steps in all workflows:

Included in protocol TWISTv2:

workflow: 'Initial QC TWIST v2' step: 'Tapestation Reception Control TWIST' aggregation step workflow: 'Library validation QC TWIST v1' step: 'Tapestation QC TWIST' aggregation step

Included in protocol Microbial WGS v5:

workflow: Microbial library validation QC v3 step: Tapestation Microbial QC aggregation step

Should we keep the old files? - Fragemnt analyzer, log and driver file? - yes for now
You will sett the qc flaggs in this step manually? - yes for now

workflow: 'Microbial Library Validation QC v3' step: 'Tapestation Microbial QC' Two times?? @Mahya-D

Included in RNA protocol:

workflow: 'Initial QC (RNA) step: 'Tapestation Reception Control (RNA)' aggregation step workflow: 'Library Validation QC (RNA)' step: 'Tapestation QC (RNA)' aggregation step

Checklist for implementation on prod:

[x] prepare_new_samples.py updated. Needs update for all reception control steps.
[x] Set up the two files in the tapestation step: Tapestation CSV and Tapestation PDF Report
[x] Add the new script for reading sizes. "Get Size (bp) From File" to the tapestation step
[x] maks sure the Size (bp) mesaurement udf is shown in tapestation step
[x] remove unnessesary udfs and buttons from tapestation step
[x] Update script for calculating average bp on prod - it should ignore NTC samples - NTC should get average

Microbial (only for protocols that did this in the tapestation step before)

[x] Activate "Set Average Size (bp)" on the aggregation step
[x] Make sure Average Size (bp) and Size (bp) udfs are shown in aggregation step
[x] Update the aggregation settings to fetch size bp
[x] Update set qc for some steps so that they are all using the same format.
[x] Chenge order of buttons
[x] remove unnessesary udfs and buttons from tapestation step

RNA (only for protocols that did this in the tapestation step before)

[x] Activate "Set Average Size (bp)" on the aggregation step
[x] Update the aggregation settings to fetch size bp
[x] Make sure Average Size (bp) and Size (bp) udfs are shown in aggregation step
[x] remove unnessesary udfs and buttons from tapestation step

Mahya-D commented 4 years ago

An example .csv file is attached here (comma-delimited).

col Sample Description: plate name_sample well (this can be changed to LIMS ID if required, however it is more practical in the lab to write plate name and source well here, and then extract the LIMS ID from that information in the script). col Average Size. Average sizes from NTC samples should not be added to LIMS.

nex191002_2019-10-03-01-HSD1000-2_compactRegionTable.pdf

annaengstrom commented 4 years ago

In the lab meeting we decided to always use Lims ID's in the tapestation files when uploading to lims using a script. If we want to change, this should be discussed in a lab meeting.

We didn't come to a conclusion in the lab meeting what we should do about the NTC's, but I guess we can exclude our in house NTC's and the routine micro's NTC's, what do you say @cesve ?

Yes, the NTC's should be excluded when calculating the average size. But these values could be removed manually after importing sizes to lims. Or perhaps we can modify the calculation script manually to remove all sizes for samples named NTC*.

Mahya-D commented 4 years ago

Suggestion for discussion in Lab meeting adding a placement step in LIMS to drag and drop 15 samples in two columns, and run Tapestation in same order without adding sample information on Agilent computer (similar to Quant-iT on Kratos). Sample information can then be retrieved from Well Id in the .csv file.

Sounds like a good idea but I'm not sure it's possible. Perhaps you can Maya? We want values for all samples but not all samples are run in the TapeStation. If we have a new placement step we would need to place all the samples.

annaengstrom commented 4 years ago

Add Lims ID's as a column in "Record Details" view in all TapeStation steps.

Mahya-D commented 4 years ago

I have discussed the possible solutions for this issue with Maya today: The decision is:

1 - In the Tape station step in LIMS, we only add the samples that we are going to measure on TapeStation to the ice bucket. The placement step will be according to well positions on Agilent tapeStation software. We do not need to add any sample information to the Software. We run and upload the pdf report and the .csv file to LIMS. @mayabrandi Maya will add a script in this step to fetch the fragment average sizes from the csv file, and she will remove the other existing LIMS scripts from this step. header in csv file for well position and size are 'WellId' and 'Average Size [bp]'. Size should not be added to LIMS for Samples with Sample name NTC* in LIMS. @mayabrandi we will need to upload two files in this step. a .pdf report file (just to be saved in LIMS), and a .csv file.

2 - We perform the Quant-iT step as before without a change.

3 - In the Aggregation step: @mayabrandi Maya will update the script to fetch sizes from step 1, and she will move the scripts from step 1 to here instead.

Mahya-D commented 4 years ago

@mayabrandi exact names in LIMS where there is a TapeStation step:

Included in protocol TWISTv2: workflow: 'Initial QC TWIST v2' step: 'Tapestation Reception Control TWIST' workflow: 'Library validation QC TWIST v1' step: 'Tapestation QC TWIST'

Included in protocol Microbial WGS v5: workflow: Microbial library validation QC v3 step: Tapestation Microbial QC workflow: 'Microbial Library Validation QC v3' step: 'Tapestation Microbial QC'

Included in RNA protocol: workflow: 'Initial QC (RNA) step: 'Tapestation Reception Control (RNA)' workflow: 'Library Validation QC (RNA)' step: 'Tapestation QC (RNA)'

mayabrandi commented 4 years ago

So the changes we discussed should be updated in all protocols where there is a tapestation step? @Mahya-D

Yes. We tried to make a list of those in the previous comment. @mayabrandi

@Mahya-D @mayabrandi we want to export and import the csv-file in all tapestation step. However, we only want to calculate the average size in the microbial protocol. We should discuss the solution in the next lab meeting on Wednesday before we make any changes.

annaengstrom commented 4 years ago

How can average size be set for all microbial samples if only some have been taken to the ice bucket?

mayabrandi commented 4 years ago

Usually we calculate the average on just a subset and the number is then applied on all the samples. Why? Someone else need to answer that. :)

11 okt. 2019 kl. 08:22 skrev Anna Engström notifications@github.com:

How can average size be set for all microbial samples if only some have been taken to the ice bucket?

Mahya-D commented 4 years ago

How can average size be set for all microbial samples if only some have been taken to the ice bucket?

@annaengstrom the new tape station step will only be for measurement on tape station. then the average sizes for all samples will be calculated in the QC aggregation step, where we add all the samples in the ice bucket. (Maya will move the script 'set average size' to the aggregation step instead.

mayabrandi commented 4 years ago

@Mahya-D Im working on this issue now. Just want to be sure you have discussed it all through and I can start working on it. Also wonder if there is an example csv file with sizes that I could work with!

Mahya-D commented 4 years ago

@Mahya-D Im working on this issue now. Just want to be sure you have discussed it all through and I can start working on it. Also wonder if there is an example csv file with sizes that I could work with!

Hi @mayabrandi . Yes, we have discussed it in the lab meeting, you can start with it. I will send you a sample .csv file via email.

mayabrandi commented 4 years ago

Thanks @Mahya-D

mayabrandi commented 4 years ago

@Mahya-D Regarding this: "Size should not be added to LIMS for Samples with Sample name NTC* in LIMS." Will those samples still go through the tapestation step? So I should in the script put an exeption for samples with NTC in the name?

mayabrandi commented 4 years ago

Will anyone from the lab be working on any of these steps on Friday? If not I will dedicate friday to put the changes into production.

annaengstrom commented 4 years ago

Friday is a good day to work on it.

Mahya-D commented 4 years ago

@Mahya-D Regarding this: "Size should not be added to LIMS for Samples with Sample name NTC* in LIMS." Will those samples still go through the tapestation step? So I should in the script put an exeption for samples with NTC in the name?

Yes, we test some of the NTCs on TapeStation, and we do not want to consider their average size in the scripts.

mayabrandi commented 4 years ago

But will the NTC samples be run on the tape station step together with samples that are not NTC? If not, then you can just skip running the scripts for those samples?

31 okt. 2019 kl. 11:15 skrev Mahya-D notifications@github.com:

@Mahya-D https://github.com/Mahya-D Regarding this: "Size should not be added to LIMS for Samples with Sample name NTC* in LIMS." Will those samples still go through the tapestation step? So I should in the script put an exeption for samples with NTC in the name?

Yes, we test some of the NTCs on TapeStation, and we do not want to consider their average size in the scripts.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Clinical-Genomics/clinical_EPPs/issues/106?email_source=notifications&email_token=AAJ65XLT6MGOX7ANYNC6UMTQRKV4TA5CNFSM4I5LSKM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECXGNSA#issuecomment-548300488, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ65XNQNM26U67RFODBBC3QRKV4TANCNFSM4I5LSKMQ.

mayabrandi commented 4 years ago

Are any of you working tomorrow? @annaengstrom @Mahya-D @annagellerbring @cesve

Mahya-D commented 4 years ago

Are any of you working tomorrow? @annaengstrom @Mahya-D @annagellerbring @cesve

I will be working tomorrow. (I will not do any labwork.) Feel free to contact me if you want to discuss anything.

Mahya-D commented 4 years ago

But will the NTC samples be run on the tape station step together with samples that are not NTC? If not, then you can just skip running the scripts for those samples?

NTCs will be run together with the samples.

Mahya-D commented 4 years ago

Hi @mayabrandi ,

little issues in Microbial Workflow.

we need a placement step, since wellID in .csv file corresponds to the Tapestation plate, and not the source plate.
the script in tape station step, and first script in aggregation step, failed. See link to the steps: https://clinical-lims.scilifelab.se/clarity/work-complete/125584 https://clinical-lims.scilifelab.se/clarity/work-complete/125585

@cesve could you please add the issue you had in Twist workflow. and probably the link to the step.

cesve commented 4 years ago

@Mahya-D I think I uploaded the wrong csv-file so I don't think there was an issue!

mayabrandi commented 4 years ago

Would you say this issue is solved now? @Mahya-D @annaengstrom @cesve

mayabrandi commented 4 years ago

Ill close this one now. Please open it again if you feel you need further help on this.

Clinical-Genomics / clinical_EPPs