Open Bontempogianpaolo1 opened 3 years ago
UPDATE:
In order to check if the software on hpc work properly I wanted to try them directly on hpc but the input files were deleted from the working group to optimize memory occupation. Thus, I am trying to install the same tools used on hpc (Arriba, EricScript and FusionCatcher) on Philae.
To install and run ARRIBA, the following commands were used (https://arriba.readthedocs.io/en/latest/quickstart/#installation-using-bioconda):
conda install -c conda-forge -c bioconda arriba=2.1.0
/home/citarrella/miniconda3/envs/Arriba2/var/lib/arriba/download_references.sh GRCh38+ENSEMBL93
run_arriba.sh STAR_index_GRCh38_ENSEMBL93/ ENSEMBL93.gtf GRCh38.fa $ARRIBA_FILES/blacklist_hg19_hs37d5_GRCh37_v2.1.0.tsv.gz $ARRIBA_FILES/known_fusions_hg19_hs37d5_GRCh37_v2.1.0.tsv.gz $ARRIBA_FILES/protein_domains_hg19_hs37d5_GRCh37_v2.1.0.gff3 8 /home/citarrella/sample/SRR064286/SRR064286_1.fastq.gz /home/citarrella/sample/SRR064286/SRR064286_2.fastq.gz
It works fine ✅
To install EricScreept, the following commands were used (https://sourceforge.net/projects/ericscript/files/):
conda install -c bioconda ericscript
chmod +x /home/citarrella/miniconda3/envs/EricScript/bin/ericscript.pl
/home/citarrella/miniconda3/envs/EricScript/bin/ericscript.pl --printdb
But after the last command, an error occurs:
❌
To install FusionCatcher the following instruction were used (https://github.com/ndaniel/fusioncatcher):
wget http://sf.net/projects/fusioncatcher/files/bootstrap.py -O bootstrap.py && python bootstrap.py --download -y
./bin/download-human_v102.sh
Now I am waiting for the end of the genome data download (90 min left).
ericScreept: try 2 things:
Final result?
I tryed to reinstall EricScript using conda install ericscript -c bioconda
but it still gives me the same error. If I try the command --help it works showing the following message:
This guy is having the same problem you have https://github.com/databio/ericscript/issues/2
So the problem could be related to ftp client...
I leave you a couple of links
If they don't work just skip it for the moment. We will find a way 😉
About FusionCatcher, at the end of the download I tryed to run the test bash script running the commands:
cd ~
/home/citarrella/fusioncatcher/test/test.sh
But it gives this error:
/home/citarrella/fusioncatcher/tools/biopython/Bio/__init__.py:128: BiopythonWarning: You may be importing Biopython from inside the source tree. This is bad practice and might lead to downstream issues. In particular, you might encounter ImportErrors due to missing compiled C extensions. We recommend that you try running your code from outside the source tree. If you are outside the source tree then you have a setup.py file in an unexpected directory: /home/citarrella/fusioncatcher/tools/biopython.
format(_parent_dir), BiopythonWarning)
Traceback (most recent call last):
File "/home/citarrella/fusioncatcher/bin/sra2illumina.py", line 48, in <module>
import phred
File "/home/citarrella/fusioncatcher/bin/phred.py", line 58, in <module>
import Bio.SeqIO
File "/home/citarrella/fusioncatcher/tools/biopython/Bio/SeqIO/__init__.py", line 387, in <module>
from Bio.Align import MultipleSeqAlignment
File "/home/citarrella/fusioncatcher/tools/biopython/Bio/Align/__init__.py", line 22, in <module>
from Bio.Align import _aligners
ImportError: cannot import name _aligners
################################################################################
################################################################################
TOTAL RUNNING TIME: 0 day(s), 0 hour(s), 0 minute(s), and 3 second(s)
################################################################################
################################################################################
sort: cannot read: test_fusioncatcher/summary_candidate_fusions.txt: No such file or directory
WARNING: Test is NOT ok! There is something wrong with FusionCatcher installation!
Here https://github.com/ndaniel/fusioncatcher/blob/master/doc/manual.md#4---installation-and-usage-examples is written:
Please, do not forget to build/download the organism data after this is done running (please notice the last lines displayed by bootstrap.py after it finished running and execute the commands suggested there, e.g. use download.sh)!
But at the end of the bootstrap.py I did not see any suggested command, so I checked the available bash scripts and there was a download-human_v102.sh file that I run, but after I got the error above. I also tried to run the build.sh file but it gives the same error.
Regarding the last comment have you already tried with conda?
About the two EricScript links you sent before, the first one refers to a 0.5.5b version that I do not find, so I tried to download the last version (0.5.5) without using Conda but I got the same error. About the second link I was trying to follow the instructions provided but I cannot find the file he suggests to delete in the first step:
Now I try to install FusionCatcher using Conda.
Would you mind open two different issues for both tools? It s becoming confusing
In the meanwhile you can use this helpful link to find files https://www.google.it/amp/s/winaero.com/find-files-linux-terminal/amp/
UPDATE:
EricScript installation:
conda create --name EricScript
conda activate EricScript
conda install -c bioconda ericscript
chmod +x /home/fcitarrella/miniconda3/envs/EricScript/bin/ericscript.pl
An error occurs running:
/home/fcitarrella/miniconda3/envs/EricScript/bin/ericscript.pl --printdb
To avoid this problem it is needed to download the EricScript database from this link and move the refid folder (ex. homo sapiens) in /home/fcitarrella/miniconda3/envs/EricScript/share/ericscript-0.5.5-5/lib/data/
✅
FusionCatcher installation:
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
conda create -n fusioncatcher fusioncatcher
source activate fusioncatcher
Before proceeding check the download-human-db.sh file in miniconda3/envs/fusioncatcher/bin/: if needed change the download version from v98 to v102 (just change the number) and the row "ln -sv current human_v102" in "ln -s human_v102 current"
Then run:
download-human-db.sh
Download fastqtk v0.27:
wget https://github.com/ndaniel/fastqtk/archive/refs/tags/v0.27.zip
unzip v0.27.zip
cd fastqtk-0.27
Make
Move "fastqtk" to fusioncatcher/bin/ directory.
It could be needed to run:
conda install tbb=2020.2
(hactar)
✅
So now there are three tools correctly installed on the server?
Exactly! Actually they are four, but I installed STAR-Fusion few weeks ago so I would like to check again if it works correctly
UPDATE:
INTEGRATE installation (https://sourceforge.net/p/integrate-fusion/wiki/Home/): Download INTEGRATE.0.2.6.tar from https://sourceforge.net/projects/integrate-fusion/files/. Run the following commands:
tar -xvf INTEGRATE.0.2.6.tar
cd INTEGRATE_0_2_6
mkdir INTEGRATE-build
cd INTEGRATE-build
conda install -c anaconda cmake
cmake ../Integrate/ -DCMAKE_BUILD_TYPE=release
make
[I tried the test and it worked. Then I tried to run a real example but the data preparation is quite complicated. I used the following command but I got some errors:
> wget ftp://ftp.ensembl.org/pub/release-75/gtf/homo_sapiens/Homo_sapiens.GRCh37.75.gtf.gz
> gzip -d Homo_sapiens.GRCh37.75.gtf.gz
> conda install -c bioconda ucsc-gtftogenepred
> gtfToGenePred -genePredExt -geneNameAsName2 Homo_sapiens.GRCh37.75.gtf Homo_sapiens.GRCh37.75.genePred
> cut -f 1-10,12 Homo_sapiens.GRCh37.75.genePred > tmp.txt
> echo -e "#GRCh37.ensGene.name\tGRCh37.ensGene.chrom\tGRCh37.ensGene.strand\tGRCh37.ensGene.txStart\tGRCh37.ensGene.txEnd\tGRCh37.ensGene.cdsStart\tGRCh37.ensGene.cdsEnd\tGRCh37.ensGene.exonCount\tGRCh37.ensGene.exonStarts\tGRCh37.ensGene.exonEnds\tGRCh37.ensemblToGeneName.value" > annot.ensembl.GRCh37.txt
> cat tmp.txt >> annot.ensembl.GRCh37.txt
> wget https://genome-idx.s3.amazonaws.com/bt/GRCh37.zip
> wget https://ccb.jhu.edu/software/tophat/downloads/tophat-2.1.1.Linux_x86_64.tar.gz
> tar -xvzf tophat-2.1.1.Linux_x86_64.tar.gz
> cd tophat-2.1.1.Linux_x86_64
> cp * /home/citarrella/miniconda3/envs/INTEGRATE/bin/conda install -c bioconda bowtie
> conda install -c bioconda bowtie2
edit the first line in your_tophat_directory/tophat from:
> tophat2 --no-coverage-search /home/citarrella/GeneFusionTools/INTEGRATE_0_2_6/preparation/GRCh37/GRCh37 /home/citarrella/sample/SRR064286/SRR064286_1.fastq,/home/citarrella/sample/SRR064286/SRR064286_2.fastq
> fastq-dump --split-files SRR15290663
> wget http://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_19/GRCh37.p13.genome.fa.gz
> gzip –d GRCh37.p13.genome.fa.gz
> conda install -c bioconda bwa
> bwa index /home/citarrella/GeneFusionTools/INTEGRATE_0_2_6/preparation/GRCh37.p13.genome.fa
> bwa mem /home/citarrella/GeneFusionTools/INTEGRATE_0_2_6/preparation/GRCh37.p13.genome.fa /home/citarrella/sample/SRR064286/SRR064286_1.fastq /home/citarrella/sample/SRR064286/SRR064286_2.fastq | samtools sort -o dna.tumor.bam – [[ERROR: invalid BAM binary header (this is not a BAM file)]]
> samtools index *.bam
> ./Integrate mkbwt /home/citarrella/GeneFusionTools/INTEGRATE_0_2_6/preparation/GRCh37.p13.genome.fa [[ERROR: the .fa file was not accepted]]
> ./Integrate fusion /home/citarrella/GeneFusionTools/INTEGRATE_0_2_6/test-data//reference.fasta annot.ensembl.GRCh37.txt ./bwts accepted_hits.bam unmapped.bam dna.tumor.bam dna.normal.bam
]
I realized a simple pipeline using EricScript. The pipeline.nf file was structured as follow:
#!/usr/bin/env nextflow
fasta1 = file(params.in1)
fasta2 = file(params.in2)
process arriba{
conda '/home/fcitarrella/miniconda3/envs/EricScript'
"""
#!/bin/bash
/home/fcitarrella/miniconda3/envs/EricScript/bin/ericscript.pl /home/fcitarrella/sample/SRR064286/SRR064286_1.fastq /home/fcitarrella/sample/SRR064286/SRR064286_2.fastq
"""
}
print "DONE"
I run it using the following instruction:
./nextflow pipeline.nf --in1 '/home/fcitarrella/sample/SRR064286/SRR064286_1.fastq' --in2 '/home/fcitarrell/sample/SRR064286/SRR064286_2.fastq'
It worked fine.
I also tried to exploit the input files but it gives some errors as shown below.
pipeline.nf:
#!/usr/bin/env nextflow
fasta1 = file(params.in1)
fasta2 = file(params.in2)
process arriba{
conda '/home/fcitarrella/miniconda3/envs/EricScript'
"""
#!/bin/bash
/home/fcitarrella/miniconda3/envs/EricScript/bin/ericscript.pl ${fasta1} ${fasta2}
"""
}
print "DONE"
error:
Error executing process > 'arriba'
Caused by:
Process `arriba` terminated with an error exit status (2)
Command executed:
#!/bin/bash
/home/fcitarrella/miniconda3/envs/EricScript/bin/ericscript.pl /home/fcitarrella/sample/SRR064286/SRR064286_1.fastq /home/fcitarrell/sample/SRR064286/SRR064286_2.fastq
Command exit status:
2
Command output:
(empty)
Command error:
Subcommands:
--checkdb Check if your database is up-to-date, based on the latest Ensembl release.
--downdb Download, build database. refid parameter need to be specified.
--simulator Generate synthetic gene fusions with the same recipe of the ericscript's paper
--calcstats Calculate the statistics that we used in our paper to evaluate the performance of the algorithms.
--------
arguments for databases subcommands (downdb, checkdb):
-db, --dbfolder <string> where database is stored. Default is ERICSCRIPT_FOLDER/lib/
--refid Genome reference identification. Run ericscript.pl --printdb to see available refid [homo_sapiens].
--printdb Print a list of available genomes and exit.
--ensversion Download data of a specific Ensembl version (>= 70). Default is the latest one.
-------
arguments for simulator:
-o, --outputfolder <string> where synthetic datasets will be stored [HOME/ericscript_simulator]
-rl, --readlength <int> length of synthetic reads [75]
--refid Genome reference identification. Run ericscript.pl --printdb to see available refid [homo_sapiens].
-v, --verbose use verbose output
--insize parameter of wgsym. Outer distance between the two ends [200]
--sd_insize parameter of wgsym. Standard deviation [50]
--ngenefusion The number of synthetic gene fusions per dataset? [50]
--min_cov Minimum coverage to simulate [1]
--max_cov Maximum coverage to simulate [50]
--nsims The number of synthetic datasets to simulate [10]
--be Use --be to generate Broken Exons (BE) data [no]
--ie Use --ie to generate Intact Exons (IE) data [yes]
-db, --dbfolder where database is stored. Default is ERICSCRIPT_FOLDER/lib/
--background_1 Fastq file (forward) for generating background reads.
--background_2 Fastq file (reverse) for generating background reads.
--nreads_background The number of reads to extract from background data [200e3].
-------
arguments for calcstats:
-o, --outputfolder <string> where statistics file will be stored [HOME/ericscript_calcstats]
--resultsfolder <string> path to folder containing algorithm results.
--datafolder <string> path to folder containing synthetic data generated by ericscript simulator.
--algoname <string> name of the algorithm that generated results.
--dataset <string> type of synthetic data to considered for calculating statistics. IE or BE?
-rl, --readlength <int> length of synthetic reads
--normroc <int> factor to normalize the score given by the algorithm.
...
Uhmm strange... have you already tried to print parameters inside that process?
There was an error in the executed command 'fcitarrell' instead of 'fcitarrella', now it works too!