X hasta

[ingkebil]

This PR rewrites the demux of hiseqx to use bcl2fastq 2.20 as well as a using a shared file system on a HPC.

• [ ] code review done by
• [x] functionality tested by @emiliaol
• [x] deploy approved by @ingkebil

This is a major:

• using a new version of bcl2fastq which has never been validated for HiSeqX
• removal of intermediate copying between servers
• creation of the output structure has been dramatically simplified based on work on Novaseq
• sourcing of correct environment has been simplified

See doc 1992:3 for test description and execution.

[ingkebil]

Nah, I don't need the approval to merge yet. I figured if anyone wanted to keep on working on it while I'm holidaying, then they could.

It isn't finished yet. Almost tho! Almost!

[ingkebil]

Alright. This works now.

What is not tested: adding of stats! (since this is not in thie repo)

[emiliaol]

Cool, I'll test it on a FC that I need demuxed :)

[emiliaol]

Testing:

Used run 181130_ST-E00269_0322_AHVCFMCCXY:

1. Copied the run to /home/proj/stage/flowcells/hiseqx/
2. Made sure the demuxstarted.txt file wasn't in the run folder
3. cloned the repo as hiseq.clinical to ~/git/
4. Used bash ~/git/demulitplexing/scripts/xcheckfornewrun.bash /home/proj/stage/flowcells/hiseqx/ /home/proj/stage/demultiplexed-runs/

First step worked fine. The jobs are submitted, waiting for them to run. See the logfile: projectlog.20190625134226.log

Failed because I had the wrong environment activated. Trying again with stage.

[emiliaol]

Tested again as hiseq.clinical. Had to make a small change to xdemuxtiles.bash where I set SLURM_ACCOUNT=production instead of development as hiseq.clinical cannot start jobs using development account.

In the stage environment was activated and the following command was run:

The jobs were submitted

The file structure was created:

And the fastq-files have been created with the correct naming and correct sizes:

The fastq-file content is also correct according to specifications:

[emiliaol]

For good measure I tested the stats adding using the current master of deliver. I added the copycomplete.txt to be able to pick the run up using checkfornewdemuxonhasta.bash:

The stats files were created in the correct place:

And they look as expected.

For some reason cg transfer flowcell does not work in the script:

But it works using the CLI directly:

So to do:

• [ ] remove the copycomplete.txt trigger from deliver and possibly add another trigger?
• [ ] find out if the cg transfer flowcell needs some work as it is used by the deliver package or find out why it failed in the script but not as a CLI command.

[ingkebil]

Nice going with the testing! :strawberry:

So, the cg transfer or anything cg won't work because checkfornewdemuxonhasta.sh doesn't source any environment. We will have to move cg from being an alias to a bash function to solve this problem. It's something trivial to implement but might be a lot of commands to test. Namely, all crontabs.

[emiliaol]

Ah, that makes sense. Can we for now add the sourcing so that we can get this in production? And then do the bash function as a separate thing?

[emiliaol]

@ingkebil @barrystokman So I started thinking about this and the scripts/hiseqx/xpostface.batch should add the copycomplete.txt so the deliver part can work. What triggers that? Should it be part of the demux process or something we will have to trigger separately? It did not happen during my test. I saw you activated prod as part of the crontab entry for something else. Would that be an option here too? So that we don't actually have to change anything in deliver?