LIONS container update

[ababaian]

Prepare LIONS Input Example [ Richard ]

• [x] Create a reference-ready ‘Fastq to Output’ container based on the first. The idea being that all reference materials (RM, hg38, etc…) are automatically downloaded and loaded directly into the container.
• [x] Adjust standard LIONS input.ctrl and parameters.ctrl example files to work with reference pipeline.
• [x] Write specific responses to each point below with how we address this [ Richard / Mehdi ]

Comments

• LIONS requires as input an RNA-seq dataset, reference genome, gene annotation and TE annotation files. It would make more sense to provide at least one set of these with the package. E.g. GRC38 + matching annotation files.
• LIONS requires a fairly long list of dependencies to be installed. This will raise the bar of use substantially, thereby potentially limiting the reach of the tool. One possibility would be to package all the dependencies in system-specific dependency packages which could be easily downloaded by end-users.
• The package uses a very old version of SAMTOOLS (0.1.18) – the current version is 1.9. They do not provide binaries for such an old version of SAMTOOLS, so users will need to build it from source, which could be difficult for some users. In general, use of such an old version could cause things to break unexpectedly since this version is no longer being maintained.
• Although there are some test data provided in the GitHub repository, no BAM file is provided. The sample input file (controls/input.list) lists a file which is not present in the GitHub repository.
• The program throws errors about missing files during runtime – this can be avoided by introducing a pre-run input check. This will greatly benefit the end-user by providing instant feedback about the locations and access permissions of input files – thereby allowing them to launch multiple jobs with confidence.

[biscuit13161]

1. A second Docker file has been provided, which will allow docker users to build a Reference-ready container, containing all the hg19 reference material and publicly available sample material. In addition a script has been provided (source_refs.sh) which will allow the downloading of hg19 and hg38 reference materials into the correct directory structure for LIONS.
2. The use of the docker container allows users to install all the required dependencies, and run LIONS, in a system-agnostic manner. The installation is relatively simple and does not require significant computational experience on the part of the user.
3. Samtools is a pre-requisite of the Bedtools package used by LIONS, the version LIONS is currently using is incompatible with the output file format used by Samtools 1.X. However the docker is configured to correctly build Samtools 0.1.18 from source. We are currently testing the newest version of bedtools with regard to this point.
4. With the introduction of the reference-ready Dockerfile, publicly available material have been provided and the input.list file has been updated to reflect these files.
5. Lions currently carries out a input check immediately prior to running the analysis. However a script (scripts/preflight_check.sh) has been added should users wish to carry out theses checks prior to running lions. Please note that many files are generated during the analysis and soft links are used to place them in the required directories; Some file systems are unable to create softlinks for these files, resulting in run-time errors. [Artem, we could do with adding a test for this and changing the execution appropriately ... I can but wasn't sure how you would want to proceed with it]

[ababaian]

Hey Richard, Sorry about the email troubles. It's a bit embarrassing our IT can't keep it operational.

The original files I've linked for the ENCODE data are quite large, since the purpose is to just get a test run online we can download a smaller sub-set of that data (same accession).

GM12878 (ENCSR000EYN)
rep1_read1: https://www.encodeproject.org/files/ENCFF000CXX/@@download/ENCFF000CXX.fastq.gz
rep1_read2: https://www.encodeproject.org/files/ENCFF000CYN/@@download/ENCFF000CYN.fastq.gz

rep2_read1: https://www.encodeproject.org/files/ENCFF000CYH/@@download/ENCFF000CYH.fastq.gz
rep2_read2: https://www.encodeproject.org/files/ENCFF000CYX/@@download/ENCFF000CYX.fastq.gz

H1esc (ENCSR000EYP)
rep1_read1: https://www.encodeproject.org/files/ENCFF000DGR/@@download/ENCFF000DGR.fastq.gz
rep1_read2: https://www.encodeproject.org/files/ENCFF000DGZ/@@download/ENCFF000DGZ.fastq.gz

rep2_read1: https://www.encodeproject.org/files/ENCFF000DGT/@@download/ENCFF000DGT.fastq.gz
rep2_read2: https://www.encodeproject.org/files/ENCFF000DHB/@@download/ENCFF000DHB.fastq.gz

K562 (ENCSR000EYO)

rep1_read1: https://www.encodeproject.org/files/ENCFF000DWV/@@download/ENCFF000DWV.fastq.gz
rep1_read2: https://www.encodeproject.org/files/ENCFF000DXN/@@download/ENCFF000DXN.fastq.gz

rep2_read1: https://www.encodeproject.org/files/ENCFF000DXE/@@download/ENCFF000DXE.fastq.gz
rep2_read2: https://www.encodeproject.org/files/ENCFF000DXW/@@download/ENCFF000DXW.fastq.gz

Hard vs. Soft Links

I forgot about that limitation about file-systems and hard links. So the issue has been that the most recent versions of tophat/cufflinks throw errors when dealing with soft links, thus I've transitioned to hard links. This creates it's own problems as you mention and I'm not certain how best to resolve this.

I think the mv operations would be one solution with obvious draw-backs.

What I'm thinking is that once everything is ready and we can test it "fastq-to-finish"

1. Use softlinks where possible and find work-around for tophat2 if it throws errors
2. Use a mix of soft/hard links to fix th2 errors, detect if file-system supports this and if not use mv to overcome any errors which arise.

Does that sound reasonable?