search

Clinical-Genomics / demultiplexing

To keep scripts associated with execution of the Illumina demultiplexing pipeline
5 stars 0 forks source link

2500 demultiplexing migration to Hasta #136

Closed barrystokman closed 3 years ago

barrystokman commented 3 years ago

@henrikstranneheim 2500 Demultiplexing use bcl2fastq 1.8.3 on thalamus, should we stick with this version or can we use the bcl2fastq 2.20 container already on hasta?

henrikstranneheim commented 3 years ago

From the documentation it says

Screenshot 2021-02-11 at 19 09 35

So I guess the question is what version of the Real-Time Analysis software are the 2500 running on. @AnnaZetterlund @AnnaLeinfelt Do you know the RTA version?

vwirta commented 3 years ago

The version of RTA might have changed during the years, which is relevant if we want to demux old FC. Looking at FC on BaseSpace, our first FC there is from March 2015. The file RunParameters.xml contains the version RTA.

<?xml version="1.0"?>
<RunParameters xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Setup>
    <ExperimentName>Pool Sig B</ExperimentName>
    <ScanID>-999</ScanID>
    <FCPosition>B</FCPosition>
    <WorkFlowType>CUSTOM</WorkFlowType>
    <PairEndFC>true</PairEndFC>
    <Read1>101</Read1>
    <IndexRead1>8</IndexRead1>
    <IndexRead2>0</IndexRead2>
    <Read2>101</Read2>
    <OutputFolder>X:\Runs</OutputFolder>
    <CompressBcls>true</CompressBcls>
    <RemapQScores>default</RemapQScores>
    <PeriodicSave>Save Tile Thumbnails</PeriodicSave>
    <Flowcell>TruSeq Rapid Flow Cell v1</Flowcell>
    <FirstBaseConfirmation>false</FirstBaseConfirmation>
    <SampleSheet>Z:\2015\HBEYJADXX.csv</SampleSheet>
    <KeepIntensityFiles>false</KeepIntensityFiles>
    <Sbs>TruSeq Rapid SBS Kit v1</Sbs>
    <Pe>TruSeq Rapid PE Cluster Kit v1</Pe>
    <Index>TruSeq Rapid PE Cluster Kit v1</Index>
    <AlignToPhiX>
      <Lane>1</Lane>
      <Lane>2</Lane>
    </AlignToPhiX>
    <ClusteringChoice>OnBoardClustering</ClusteringChoice>
    <RapidRunChemistry>RapidRunV1</RapidRunChemistry>
    <RunMode>RapidRun</RunMode>
    <Rehyb>None</Rehyb>
    <PerformPreRunFluidicsCheck>false</PerformPreRunFluidicsCheck>
    <ServiceRun>false</ServiceRun>
    <ApplicationName>HiSeq Control Software</ApplicationName>
    <ApplicationVersion>2.2.58</ApplicationVersion>
    <RunID>150306_D00410_0144_BHBEYJADXX</RunID>
    <RunStartDate>150306</RunStartDate>
    <IntegrationMode>BaseSpace</IntegrationMode>
    <BaseSpaceSettings>
      <RunMonitoringOnly>true</RunMonitoringOnly>
      <Username>clinical.genomics@scilifelab.se</Username>
      <PlannedRun>false</PlannedRun>
      <RunId>11127149</RunId>
      <TempFolder>E:\Illumina\BaseSpaceTemp</TempFolder>
      <SendInstrumentHealthToILMN>true</SendInstrumentHealthToILMN>
    </BaseSpaceSettings>
    <ScannerID>D00410</ScannerID>
    <ScanNumber>144</ScanNumber>
    <ComputerName>HWI-D00410</ComputerName>
    <FPGAVersion>7.9.7</FPGAVersion>
    <CPLDVersion>3.0.0</CPLDVersion>
    <RTAVersion>1.18.64</RTAVersion>
    <ChemistryVersion>Illumina,Bruno Fluidics Controller,0,v2.0340</ChemistryVersion>
    <CameraFirmware>2.01-F20-R02</CameraFirmware>
    <CameraDriver>6.45.20.3690</CameraDriver>
    <FocusCameraFirmware />
    <Barcode>HBEYJADXX</Barcode>
    <Username>sbsuser</Username>
    <SelectedSections>
      <Section Name="A_1" />
      <Section Name="B_1" />
    </SelectedSections>
    <FocusMethod>DynamicITF</FocusMethod>
    <SelectedSurface>BothLaneSurfaces</SelectedSurface>
    <SwathScanMode>DualSwathFC</SwathScanMode>
    <EnableLft>true</EnableLft>
    <AutoTiltOnce>true</AutoTiltOnce>
    <EnableAutoCenter>true</EnableAutoCenter>
    <EnableAnalysis>true</EnableAnalysis>
    <EnableBasecalling>true</EnableBasecalling>
    <EnableCameraLogging>false</EnableCameraLogging>
    <AdapterPlate>HiSeq Adapter Plate</AdapterPlate>
    <SlideHolder>HiSeq Flow Cell Holder</SlideHolder>
    <TemplateCycleCount>5</TemplateCycleCount>
    <NumAnalysisThreads>8</NumAnalysisThreads>
    <FPGADynamicFocusSettings>
      <MaxInitialZJumpHalfUm>3</MaxInitialZJumpHalfUm>
      <MaxSubsequentZJumpHalfUm>7</MaxSubsequentZJumpHalfUm>
      <NumberOfInitialZJumps>0</NumberOfInitialZJumps>
      <CVGainStart>500</CVGainStart>
      <CVGainPosLocked>500</CVGainPosLocked>
      <Offset>250</Offset>
      <HotPixel>350</HotPixel>
      <MotorDelayFrames>10</MotorDelayFrames>
      <SoftwareLaserLag>200</SoftwareLaserLag>
      <DitherSize>100</DitherSize>
      <GroupSize>50</GroupSize>
      <DitherShift>0</DitherShift>
      <IntensityCeiling>65535</IntensityCeiling>
      <IGain>100</IGain>
      <IHistory>4</IHistory>
    </FPGADynamicFocusSettings>
    <TileWidth>2048</TileWidth>
    <TileHeight>10000</TileHeight>
    <ImageWidth>2048</ImageWidth>
    <ImageHeight>160000</ImageHeight>
    <AreaPerPixelmm2>1.40625E-07</AreaPerPixelmm2>
    <LaneLength>60</LaneLength>
    <NumTilesPerSwath>16</NumTilesPerSwath>
    <NumSwaths>2</NumSwaths>
    <UseExistingRecipe>false</UseExistingRecipe>
    <Reads>
      <Read Number="1" NumCycles="101" IsIndexedRead="N" />
      <Read Number="2" NumCycles="8" IsIndexedRead="Y" />
      <Read Number="3" NumCycles="101" IsIndexedRead="N" />
    </Reads>
    <EnableNotifications>false</EnableNotifications>
    <ReagentKits>
      <Sbs>
        <SbsReagentKit>
          <ID>x</ID>
          <Prime>false</Prime>
          <NumberCyclesRemaining>225</NumberCyclesRemaining>
          <IsNew50Cycle>false</IsNew50Cycle>
          <IsNew200Cycle>true</IsNew200Cycle>
          <IsNew500Cycle>false</IsNew500Cycle>
        </SbsReagentKit>
      </Sbs>
      <Index>
        <ReagentKit>
          <ID>x</ID>
        </ReagentKit>
      </Index>
      <Pe />
      <Rehyb />
    </ReagentKits>
    <ReagentBottles>
      <Sbs />
    </ReagentBottles>
    <Resume>false</Resume>
    <ResumeCycle>0</ResumeCycle>
    <SupportMultipleSurfacesInUI>true</SupportMultipleSurfacesInUI>
    <TempFolder>E:\Illumina\HiSeqTemp\150306_D00410_0144_BHBEYJADXX</TempFolder>
    <RecipeFragmentVersion>1.5.21.0</RecipeFragmentVersion>
    <PromptForPeReagents>false</PromptForPeReagents>
    <MockRun>false</MockRun>
    <ScannedBarcode />
  </Setup>
  <Version>1</Version>
</RunParameters>

Googling I can only find info from July 2014, when RTA v1.18.61 was released.

Our first FC is from October 2013, but according to statusDB the first one that we have on disk is H82C9ADXX I guess the RunParameters.xml file is also on disk...

AnnaLeinfelt commented 3 years ago

From the documentation it says

Screenshot 2021-02-11 at 19 09 35

So I guess the question is what version of the Real-Time Analysis software are the 2500 running on. @AnnaZetterlund @AnnaLeinfelt Do you know the RTA version?

We are running RTA 1.18.64 Info from sequencing logbook, https://docs.google.com/spreadsheets/d/1viDi0Wvbn-jRsleUhDV3c34JTCItVqRxIZYPFNS0FPQ/edit#gid=11

henrikstranneheim commented 3 years ago

Thank for the input!

It would be great to only have to use bcl2fastq version 2.20 for all demultiplexing processes both retrospective and prospective.

Based on the input we should probably use bcl2fastq version 2.20 for the 2500 demultiplexing done today and not bcl2fastq version 1.8.3.

I think there are some questions we should try to answer:

AnnaLeinfelt commented 3 years ago

Thank for the input!

It would be great to only have to use bcl2fastq version 2.20 for all demultiplexing processes both retrospective and prospective.

Based on the input we should probably use bcl2fastq version 2.20 for the 2500 demultiplexing done today and not bcl2fastq version 1.8.3.

I think there are some questions we should try to answer:

  • Have we ever performed any sequencing with an RTA version less than 1.18.54?

From the old edition of the HiSeq instrument manual i found this

2014-03-18 Real Time Analysis—The integrated primary analysis software, Real Time Analysis (RTA), performs base calling and assigns a quality score to each base for each cycle. Current version 1.17.21.3.

2015-03-10 Real Time Analysis—The integrated primary analysis software, Real Time Analysis (RTA), performs base calling and assigns a quality score to each base for each cycle. Two different versions of RTA are in use, 1.18.61 and 1.18.64.

So there seems to be a period in 2014 where an RTA version of 1.17.xx was used. An educated guess is that the RTA version was upgraded sometime in between autumn 2014 and Februari 2015. I can not seem to find an exact date.

In April 2015 we validated the HiSeq X runs.

  • Do we need to verify 2500 demultiplexing using bcl2fastq version 2.20?

    • Based on the documentation it actually seems more correct to use bcl2fastq version 2.20!

Today we are only running the NIPT on the HiSeq2500 (and that application is planned to use the NovaSeq instead), RML samples for the single cell facility and there has been a couple of RML samples for a research group in Alfa 3. When NIPT is running on NovaSeq there is no plan to use the HiSeq 2500s anymore. So there is no need to verify bcl2fastq version 2.20 for future runs as far as I see, the only reason should be for re-runs.

mayabrandi commented 3 years ago

paths in lims to concider before moving to hasta 

java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.1 '/srv/mfs/hiseq_data/gls_events/gls_events_arnold'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.2 '/srv/mfs/hiseq_data/gls_events/gls_events_deagol'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.3 '/srv/mfs/hiseq_data/gls_events/gls_events_eomer'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.4 '/srv/mfs/hiseq_data/gls_events/gls_events_sylvester'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.5 '/srv/mfs/hiseq_data/gls_events/gls_events_merida'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.6 '/srv/mfs/hiseq_data/gls_events/gls_events_sigourney'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.7 '/srv/mfs/hiseq_data/gls_events/gls_events_tiamat'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.8 '/srv/mfs/hiseq_data/gls_events/gls_events_ishtar'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectory.99 '/opt/gls/clarity/smoketests/hiseq/monitored'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirectorySuffixes '5,6'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.1 '1'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.2 '2'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.3 '3'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.4 '4'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.5 '5'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.6 '6'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.7 '7'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.eventFileDirsToUseForSearchReplacePair.8 '8'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.ignoreUnmatchedContainerIds 'true'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.ignoreUnmatchedContainerIdsWaitDays '14'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.1 '/home/gls/hiseq_data/Runs_seq-nas-3/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.2 '/home/gls/hiseq_data/Runs_nas-6/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.3 '/home/gls/hiseq_data/Runs_nas-6/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.4 '/home/gls/hiseq_data/Runs_clinical-nas-2/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.5 '/home/gls/hiseq_data/Runs_seq-nas-3/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.6 '/home/gls/hiseq_data/Runs_clinical-nas-1/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.7 '/home/gls/hiseq_data/Runs_seq-nas-1/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.8 '/home/gls/hiseq_data/Runs_seq-nas-3/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixReplace.99 '/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.1 'X:\Runs\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.2 'X:\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.3 'X:\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.4 'X:\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.5 'X:\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.6 'X:\Runs\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.7 'X:\Runs\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.8 'X:\Runs\'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearch.99 '/'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.netPathPrefixSearchReplaceSuffixes '1,5,6,8'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.runReportViewsVersion '4'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.sequenceProcessBaseName 'CG002 - Illumina Sequencing (Illumina SBS)'
java -jar omxprops-ConfigTool.jar set -y -f clinical-lims.scilifelab.se  hiseqga.seqservice.synchronizationPeriod '60'
henrikstranneheim commented 3 years ago

@barrystokman are we ready to close this one?

barrystokman commented 3 years ago

Almost, I found a mistake in the nipt viewer cronjob, fix here: https://github.com/Clinical-Genomics/servers/pull/513

barrystokman commented 3 years ago

Then, I would like to monitor the NITP viewer until it's verified that it actually works, due to the precarious situation of the NIPT viewer code...

barrystokman commented 3 years ago

The NIPT viewer works, as was confirmed today. Very last step is figuring out how to handle the CRONIC emails, because now we will get spammed even though it;s working as intended.

barrystokman commented 3 years ago

Correction: the emails are handled by the cron daemon, not CRONIC.

barrystokman commented 3 years ago

PR to stop emails spam here: https://github.com/Clinical-Genomics/servers/pull/515