Closed Caro-Ca closed 4 years ago
Yes please provide unzipped fastq as input. According to the stdout the bwa mem
step didn't finish successfully, which only takes unzipped fastq.
Thanks for your reply; however, I still have the error even though I already gunzip the files.
[bwa_index] Pack FASTA... 0.10 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 4.04 seconds elapse.
[bwa_index] Update BWT... 0.08 sec
[bwa_index] Pack forward-only FASTA... 0.04 sec
[bwa_index] Construct SA from BWT and Occ... 1.18 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index -p /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta
[main] Real time: 5.760 sec; CPU: 5.445 sec
mkdir: cannot create directory ?/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper?: File exists
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 36 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq
[main] Real time: 0.002 sec; CPU: 0.003 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 36 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
[main] Real time: 0.002 sec; CPU: 0.002 sec
Error in aLength:max(lengths) : result would be too long a vector
Calls: GetSamFile
In addition: Warning message:
In max(lengths) : no non-missing arguments to max; returning -Inf
Execution halted
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/bed_tsd/SRR4209894_paired_1_SRR4209894_paired_2insertions.bed' for reading (No such file or directory)
Error: The requested bed file (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/SRR4209894_paired_ngs_te_mapper_presort.bed) could not be opened. Exiting!
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/bed_tsd/*.bed': No such file or directory
Running PoPoolationTE pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq' for reading (No such file or directory)
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq' for reading (No such file or directory)
[main] Version: 0.7.4-r385
[main] CMD: bwa bwasw -t 36 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/reads1.fastq
[main] Real time: 0.027 sec; CPU: 0.027 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa bwasw -t 36 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/reads2.fastq
[main] Real time: 0.029 sec; CPU: 0.027 sec
Median insert size =
runpopoolationte.sh: line 41: * 3 + -1: syntax error: operand expected (error token is "* 3 + -1")
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/SRR4209894_paired_popoolationte*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/te-poly-filtered.txt': No such file or directory
unzip: cannot find or open *.zip, *.zip.zip or *.zip.ZIP.
No zipfiles found.
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/results/summary/fastqc_analysis/*_1*/fastqc_data.txt': No such file or directory
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/results/summary/fastqc_analysis/*_2*/fastqc_data.txt': No such file or directory
Can you provide the command you used to run McClintock?
This is my python script:
#!/usr/bin/env python
import os
import subprocess
if __name__=='__main__':
path = os.getcwd()
dir_files = os.listdir(path)
pair_reads = {}
for file in sorted(dir_files):
if file.endswith("_paired_1.fastq"):
file1 = file
if file.endswith("_paired_2.fastq"):
file2 = file
pair_reads[file1] = file2
for key, value in pair_reads.items():
code = "bash ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_te_families.tsv -1 {} -2 {} -o /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results -p 36".format(key, value)
cmd = subprocess.check_output(code, shell =True)
print( "EXIT STATUS AND TYPE", cmd)
The SRR4210051_paired_2.fastq are the files after a trimming process of the adapters. This is a set of files in my directory:
-rw-r--r-- 1 silviav binf 171921028 Jan 31 16:35 SRR4210051.bam
-rw-r--r-- 1 silviav binf 37312 Jan 31 16:35 SRR4210051.bam.bai
-rw-r--r-- 1 silviav binf 425302 Feb 3 10:04 SRR4210051.split.bam
-rw-r--r-- 1 silviav binf 39765824 Jan 31 16:35 SRR4210051.umap.fastq
lrwxrwxrwx 1 silviav binf 74 Dec 19 15:48 SRR4210051_1.fastq.gz -> /hosts/linuxhome/chaperone/michael/Kitaetal2017_eQTL/SRR4210051_1.fastq.gz
lrwxrwxrwx 1 silviav binf 74 Dec 19 15:48 SRR4210051_2.fastq.gz -> /hosts/linuxhome/chaperone/michael/Kitaetal2017_eQTL/SRR4210051_2.fastq.gz
-rw-r--r-- 1 silviav binf 439232192 Jan 31 15:25 SRR4210051_paired_1.fastq
-rw-r--r-- 1 silviav binf 439920604 Jan 31 15:25 SRR4210051_paired_2.fastq
SRR4210051_paired_1.fastq
and SRR4210051_paired_2.fastq
are correctly paired), but you didn't provide paired end data to McClintock correctly. If you plan to run McClintock in pair end mode, you need to provide read 1 with -1
and read 2 with -2
(see example below).
bash ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_te_families.tsv -1 SRR4210051_paired_1.fastq -2 SRR4210051_paired_2.fastq
Yes, indeed I'm running paired data, but I'm assigning -1 as a key and -2 as a value in my dictionary. bash ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_te_families.tsv -1 {} -2 {} -o /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results -p 36".format(key, value)
My dataset has more than 1x10^3 files so that is why I'm making a python script to run each pair per folder. Thank you in advance for your help.
On the stdout says that SRR4209894_paired_1.fastq
doesn't exist but it does.
The entire stdout message is:
:python McClintock_Kita.py
Running McClintock version: 1d06a94bf7bcd5a1e17dfc8336fb82895780c713
Date of run is 23_02_20
Creating directory structure...
cp: cannot stat 'SRR4209894_paired_1.fastq': No such file or directory
cp: cannot stat 'SRR4209894_paired_2.fastq': No such file or directory
Performing FastQC analysis...
Skipping '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq' which didn't exist, or couldn't be read
Skipping '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq' which didn't exist, or couldn't be read
[bwa_index] Pack FASTA... 0.09 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 4.01 seconds elapse.
[bwa_index] Update BWT... 0.08 sec
[bwa_index] Pack forward-only FASTA... 0.04 sec
[bwa_index] Construct SA from BWT and Occ... 1.19 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta
[main] Real time: 5.760 sec; CPU: 5.412 sec
[bwa_index] Pack FASTA... 0.10 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 4.30 seconds elapse.
[bwa_index] Update BWT... 0.08 sec
[bwa_index] Pack forward-only FASTA... 0.05 sec
[bwa_index] Construct SA from BWT and Occ... 1.28 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta
[main] Real time: 6.121 sec; CPU: 5.811 sec
Creating sam alignment...
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 24 -v 0 -R @RG\tID:SRR4209894_paired\tSM:SRR4209894_paired /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
[main] Real time: 0.022 sec; CPU: 0.021 sec
Calculating median insert size...
Median insert size =
Creating bam alignment files...
[bam_header_read] EOF marker is absent. The input is probably truncated.
[samopen] SAM header is present: 18 sequences.
[sam_read1] reference 'ID:SRR4209894_paired SM:SRR4209894_paired
' is recognized as '*'.
[main_samview] truncated file.
Sorting sam alignment...
Running TE-locate pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
runtelocate.sh: line 12: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired * 5: syntax error: operand expected (error token is "/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired * 5")
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TE-locate/SRR4209894_paired_telocate*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TE-locate/*.info': No such file or directory
Running RetroSeq pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
Error: The requested bed file (4639.discordant_mates.bed) could not be opened. Exiting!
Error: The requested bed file (4639.discordant_mates.bed) could not be opened. Exiting!
Error: The requested bed file (4639.discordant_mates.bed) could not be opened. Exiting!
Error: The requested bed file (4639.discordant_mates.bed) could not be opened. Exiting!
Error: The requested bed file (4639.discordant_mates.bed) could not be opened. Exiting!
Error: The requested bed file (4639.discordant_mates.bed) could not be opened. Exiting!
Error: The requested bed file (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RetroSeq/SRR4209894_paired_retroseq_presort.txt) could not be opened. Exiting!
Error: The requested bed file (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RetroSeq/tmp) could not be opened. Exiting!
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
Running TEMP pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TEMP/SRR4209894_paired_temp*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TEMP/SRR4209894_paired.insertion.refined.bp.summary': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TEMP/SRR4209894_paired.absence.refined.bp.summary': No such file or directory
Running RelocaTE pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
use -h option to get help
runrelocate.sh: line 37: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_redundant.bed: No such file or directory
runrelocate.sh: line 38: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_nonredundant.bed: No such file or directory
runrelocate.sh: line 43: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_ref_presort_redundant.txt: No such file or directory
runrelocate.sh: line 44: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_presort_redundant.txt: No such file or directory
runrelocate.sh: line 48: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_presort_redundant.txt: No such file or directory
runrelocate.sh: line 50: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_precut_redundant.txt: No such file or directory
runrelocate.sh: line 51: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_counted_precut_redundant.txt: No such file or directory
runrelocate.sh: line 53: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_redundant.bed: No such file or directory
runrelocate.sh: line 56: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/tmp: No such file or directory
sort: cannot read: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_counted_precut_redundant.txt: No such file or directory
runrelocate.sh: line 57: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_nonredundant.bed: No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_counted_precut_redundant.txt': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_precut_redundant.txt': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_presort_redundant.txt': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/tmp': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_ref_presort_redundant.txt': No such file or directory
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/*/results': No such file or directory
Running ngs_te_mapper pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
[bwa_index] Pack FASTA... 0.00 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 0.01 seconds elapse.
[bwa_index] Update BWT... 0.00 sec
[bwa_index] Pack forward-only FASTA... 0.00 sec
[bwa_index] Construct SA from BWT and Occ... 0.00 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index -p /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs.fasta
[main] Real time: 0.021 sec; CPU: 0.014 sec
[bwa_index] Pack FASTA... 0.10 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 4.00 seconds elapse.
[bwa_index] Update BWT... 0.08 sec
[bwa_index] Pack forward-only FASTA... 0.04 sec
[bwa_index] Construct SA from BWT and Occ... 1.16 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index -p /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta
[main] Real time: 5.684 sec; CPU: 5.379 sec
mkdir: cannot create directory ‘/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper’: File exists
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq
[main] Real time: 0.002 sec; CPU: 0.002 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
[main] Real time: 0.002 sec; CPU: 0.002 sec
Error in aLength:max(lengths) : result would be too long a vector
Calls: GetSamFile
In addition: Warning message:
In max(lengths) : no non-missing arguments to max; returning -Inf
Execution halted
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/bed_tsd/SRR4209894_paired_1_SRR4209894_paired_2insertions.bed' for reading (No such file or directory)
Error: The requested bed file (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/SRR4209894_paired_ngs_te_mapper_presort.bed) could not be opened. Exiting!
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/bed_tsd/*.bed': No such file or directory
Running PoPoolationTE pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq' for reading (No such file or directory)
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq' for reading (No such file or directory)
[main] Version: 0.7.4-r385
[main] CMD: bwa bwasw -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/reads1.fastq
[main] Real time: 0.026 sec; CPU: 0.026 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa bwasw -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/reads2.fastq
[main] Real time: 0.026 sec; CPU: 0.027 sec
Median insert size =
runpopoolationte.sh: line 41: * 3 + -1: syntax error: operand expected (error token is "* 3 + -1")
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/SRR4209894_paired_popoolationte*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/te-poly-filtered.txt': No such file or directory
unzip: cannot find or open *.zip, *.zip.zip or *.zip.ZIP.
No zipfiles found.
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/results/summary/fastqc_analysis/*_1*/fastqc_data.txt': No such file or directory
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/results/summary/fastqc_analysis/*_2*/fastqc_data.txt': No such file or directory
mcclintock.sh
for one sample only, provide the stdout and output directory for the same sample)/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq
and /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
don't exist. Can you confirm whether you provide the correct file path?Please do ls -lsrth /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq
ls -lsrth /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
to help me confirm the read 1 and read 2 for SRR4209894 exist and is readable.
The directory for the parameter -o is empty, I created it to get the results.
ls -lsrth
/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq': No such file or directory
silviav@mutant9:/hosts/linuxhome/chaperone/silviav/reads/Kita:ls -lsrth /hosts/linuxhome/chaperone/silviav/reads/Kita/SRR4209894_paired_1.fastq
391M -rw-r--r-- 1 silviav binf 391M Jan 31 14:11 /hosts/linuxhome/chaperone/silviav/reads/Kita/SRR4209894_paired_1.fastq
SRR4209894_paired_1.fastq
does exist, so what's the complete path for this file?mcclintock.sh
for SRR4209894? You can print the command from your python script.There are three directories:
1) Where the -1 and -2 files are located and where I run the code /hosts/linuxhome/chaperone/silviav/reads/Kita/
2) an empty folder that I created to get the output of McClintock run called /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results
3) Where the -r, -c, -g, -t files are located (~/mcclintock/test/
)
The file SRR4209894_paired_1.fastq exists in 1) but not in 2)
I have the feeling the problem is the parameter -o which is an empty folder. However, this is the stout when I run mcclinctock:
bash ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_te_families.tsv -1 SRR4209894_paired_1.fastq -2 SRR4209894_paired_2.fastq -o /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results -p 24
Running McClintock version:
Running McClintock version: 1d06a94bf7bcd5a1e17dfc8336fb82895780c713
1d06a94bf7bcd5a1e17dfc8336fb82895780c713
Date of run is 23_02_20
Date of run is 23_02_20
Creating directory structure...
Creating directory structure...
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
cp: cannot stat 'SRR4209894_paired_1.fastq': No such file or directory
cp: cannot stat 'SRR4209894_paired_2.fastq': No such file or directory
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Performing FastQC analysis...
Performing FastQC analysis...
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Skipping '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq' which didn't exist, or couldn't be read
Skipping '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq' which didn't exist, or couldn't be read
[bwa_index] Pack FASTA... 0.10 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 4.01 seconds elapse.
[bwa_index] Update BWT... 0.08 sec
[bwa_index] Pack forward-only FASTA... 0.04 sec
[bwa_index] Construct SA from BWT and Occ... 1.21 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta
[main] Real time: 5.727 sec; CPU: 5.435 sec
[bwa_index] Pack FASTA... 0.10 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 4.28 seconds elapse.
[bwa_index] Update BWT... 0.08 sec
[bwa_index] Pack forward-only FASTA... 0.05 sec
[bwa_index] Construct SA from BWT and Occ... 1.28 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta
[main] Real time: 6.098 sec; CPU: 5.792 sec
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Creating sam alignment...
Creating sam alignment...
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 24 -v 0 -R @RG\tID:SRR4209894_paired\tSM:SRR4209894_paired /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
[main] Real time: 0.021 sec; CPU: 0.021 sec
Calculating median insert size...
Calculating median insert size...
Median insert size =
Median insert size =
Creating bam alignment files...
Creating bam alignment files...
[bam_header_read] EOF marker is absent. The input is probably truncated.
[samopen] SAM header is present: 18 sequences.
[sam_read1] reference 'ID:SRR4209894_paired SM:SRR4209894_paired
' is recognized as '*'.
[main_samview] truncated file.
Sorting sam alignment...
Sorting sam alignment...
Running TE-locate pipeline...
Running TE-locate pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
runtelocate.sh: line 12: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired * 5: syntax error: operand expected (error token is "/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired * 5")
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TE-locate/SRR4209894_paired_telocate*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TE-locate/*.info': No such file or directory
Running RetroSeq pipeline...
Running RetroSeq pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
RetroSeq: A tool for discovery and genotyping of transposable elements from short read alignments
Version: 1.41
Author: Thomas Keane (thomas.keane@sanger.ac.uk)
Reading -refTEs file: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/../reference/sacCer2_locationlist
Min anchor quality: 20
Min percent identity: 80
Min length for hit: 36
Opening BAM (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/bam/SRR4209894_paired.bam) and getting initial set of candidate mates....
0 candidate reads remain to be found after first pass....
Using reference TE locations to assign discordant mates...
Screening for hits to: TY4
Error: The requested bed file (9074.discordant_mates.bed) could not be opened. Exiting!
Screening for hits to: TY2
Error: The requested bed file (9074.discordant_mates.bed) could not be opened. Exiting!
Screening for hits to: TY3_1p
Error: The requested bed file (9074.discordant_mates.bed) could not be opened. Exiting!
Screening for hits to: TY5
Error: The requested bed file (9074.discordant_mates.bed) could not be opened. Exiting!
Screening for hits to: TY1
Error: The requested bed file (9074.discordant_mates.bed) could not be opened. Exiting!
Screening for hits to: TY3
Error: The requested bed file (9074.discordant_mates.bed) could not be opened. Exiting!
RetroSeq finished successfully
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
RetroSeq: A tool for discovery and genotyping of transposable elements from short read alignments
Version: 1.41
Author: Thomas Keane (thomas.keane@sanger.ac.uk)
Found sample: SRR4209894_paired
Calling sample SRR4209894_paired
Beginning paired-end calling...
Creating VCF file of calls....
Error: The requested bed file (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RetroSeq/SRR4209894_paired_retroseq_presort.txt) could not be opened. Exiting!
Error: The requested bed file (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RetroSeq/tmp) could not be opened. Exiting!
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
Running TEMP pipeline...
Running TEMP pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
Supply a bam file created by bwa mem as option 1
Supply a reference 2bit file as option 2
Supply a consensus TE sequence file (fasta) as option 3
Supply reference insertion locations in bed format as option 4
Supply TE hierarchy file as option 5
Supply the median insert length as option 6
Supply the sample name as option 7
Supply the number of processors to use as option 8
Supply output folder as option 9
Supply reference insertion locations in gff format with correct family names as the ID as option 10
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TEMP/SRR4209894_paired_temp*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TEMP/SRR4209894_paired.insertion.refined.bp.summary': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/TEMP/SRR4209894_paired.absence.refined.bp.summary': No such file or directory
Running RelocaTE pipeline...
Running RelocaTE pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Must provide at least 1 short read file
use -h option to get help
runrelocate.sh: line 37: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_redundant.bed: No such file or directory
runrelocate.sh: line 38: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_nonredundant.bed: No such file or directory
runrelocate.sh: line 43: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_ref_presort_redundant.txt: No such file or directory
runrelocate.sh: line 44: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_presort_redundant.txt: No such file or directory
runrelocate.sh: line 48: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_presort_redundant.txt: No such file or directory
runrelocate.sh: line 50: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_precut_redundant.txt: No such file or directory
runrelocate.sh: line 51: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_counted_precut_redundant.txt: No such file or directory
runrelocate.sh: line 53: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_redundant.bed: No such file or directory
runrelocate.sh: line 56: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/tmp: No such file or directory
sort: cannot read: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_counted_precut_redundant.txt: No such file or directory
runrelocate.sh: line 57: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_nonredundant.bed: No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_counted_precut_redundant.txt': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_precut_redundant.txt': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_presort_redundant.txt': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/tmp': No such file or directory
rm: cannot remove '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate_ref_presort_redundant.txt': No such file or directory
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/SRR4209894_paired_relocate*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/RelocaTE/*/results': No such file or directory
Running ngs_te_mapper pipeline...
Running ngs_te_mapper pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
During startup - Warning message:
Setting LC_CTYPE failed, using "C"
R version 3.5.0 (2018-04-23)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Ubuntu 16.04.6 LTS
Matrix products: default
BLAS/LAPACK: /home/silviav/miniconda/envs/MCCLINTOCK_ngstemapper/lib/R/lib/libRblas.so
locale:
[1] LC_CTYPE=C LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.5.0
[1] "sample=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq;/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq"
[2] "genome=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta"
[3] "teFile=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs.fasta"
[4] "tsd=20"
[5] "thread=24"
[6] "output=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper"
[7] "sourceCodeFolder=/home/silviav/mcclintock/ngs_te_mapper/sourceCode"
going to analyse: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq;/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
in to: /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper
with
maximum size of TSD: 20
number of thread used: 24
[1] "getting the bwa index for the TE set"
[bwa_index] Pack FASTA... 0.00 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 0.01 seconds elapse.
[bwa_index] Update BWT... 0.00 sec
[bwa_index] Pack forward-only FASTA... 0.00 sec
[bwa_index] Construct SA from BWT and Occ... 0.00 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index -p /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs.fasta
[main] Real time: 0.033 sec; CPU: 0.015 sec
[1] "getting the bwa index for the Genome"
[bwa_index] Pack FASTA... 0.10 sec
[bwa_index] Construct BWT for the packed sequence...
[bwa_index] 4.00 seconds elapse.
[bwa_index] Update BWT... 0.08 sec
[bwa_index] Pack forward-only FASTA... 0.04 sec
[bwa_index] Construct SA from BWT and Occ... 1.16 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa index -p /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta
[main] Real time: 5.686 sec; CPU: 5.375 sec
mkdir: cannot create directory '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper': File exists
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq
[main] Real time: 0.003 sec; CPU: 0.003 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa mem -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sac_cer_TE_seqs /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq
[main] Real time: 0.002 sec; CPU: 0.002 sec
Error in aLength:max(lengths) : result would be too long a vector
Calls: GetSamFile
In addition: Warning message:
In max(lengths) : no non-missing arguments to max; returning -Inf
Execution halted
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/bed_tsd/SRR4209894_paired_1_SRR4209894_paired_2insertions.bed' for reading (No such file or directory)
Error: The requested bed file (/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/SRR4209894_paired_ngs_te_mapper_presort.bed) could not be opened. Exiting!
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/bed_tsd/*.bed': No such file or directory
Running PoPoolationTE pipeline...
Running PoPoolationTE pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_1.fastq' for reading (No such file or directory)
awk: fatal: cannot open file `/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/reads/SRR4209894_paired_2.fastq' for reading (No such file or directory)
[main] Version: 0.7.4-r385
[main] CMD: bwa bwasw -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/reads1.fastq
[main] Real time: 0.027 sec; CPU: 0.027 sec
[main] Version: 0.7.4-r385
[main] CMD: bwa bwasw -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/reads2.fastq
[main] Real time: 0.027 sec; CPU: 0.027 sec
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Median insert size =
Median insert size =
runpopoolationte.sh: line 41: * 3 + -1: syntax error: operand expected (error token is "* 3 + -1")
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/SRR4209894_paired_popoolationte*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/te-poly-filtered.txt': No such file or directory
unzip: cannot find or open *.zip, *.zip.zip or *.zip.ZIP.
No zipfiles found.
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/results/summary/fastqc_analysis/*_1*/fastqc_data.txt': No such file or directory
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/results/summary/fastqc_analysis/*_2*/fastqc_data.txt': No such file or directory
/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results
is empty, did you mean it doesn't exist? I would be suspious if it doesn't exist since McClintock doesn't produce any error message when creating the directory structure.bash ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_te_families.tsv -1 SRR4209894_paired_1.fastq -2 SRR4209894_paired_2.fastq -o /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results -p 24
directly in bash. When you try that, please provide complete path for read 1 and read 2.mcclintock.sh
, then it's possible McClintock failed to create out directory and create symbolic link in your run, which could be due to issues in your environment.Hi again,
This folder I created/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results
but it doesn't contain any files only when I ran mcclintocksacCer2
is created.
On the other hand, I ran the code where the bash is and I don't get any errors or at least is not the same stdout as before. This is a piece of the stdout:
finished the job and found 4 new insertions run:
sourceCode/ngs_te_logo.R genome=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/sacCer2.fasta output=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/logo inputFolder=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/metadata outputFile=/hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/ngs_te_mapper/allSamples.bed window=25
to get the logos centred at the TSD with +/- 25 bp to both sides
also found 4 previously known insertions
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
Running PoPoolationTE pipeline...
Running PoPoolationTE pipeline...
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
[bsw2_aln] read 1187751 sequences/pairs (118431494 bp) ...
[main] Version: 0.7.4-r385
[main] CMD: bwa bwasw -t 24 /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/reference/popoolationte_full_sacCer2.fasta /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results/sacCer2/SRR4209894_paired/PoPoolationTE/reads1.fastq
[main] Real time: 19.464 sec; CPU: 338.075 sec
[bsw2_aln] read 1187751 sequences/pairs (118887517 bp) ...
2 archives were successfully processed.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_PAPER = "en_US.UTF-8",
LC_ADDRESS = "en_US.UTF-8",
LC_MONETARY = "en_US.UTF-8",
LC_NUMERIC = "en_US.UTF-8",
LC_TELEPHONE = "en_US.UTF-8",
LC_IDENTIFICATION = "en_US.UTF-8",
LC_MEASUREMENT = "en_US.UTF-8",
LC_CTYPE = "UTF-8",
LC_TIME = "en_US.UTF-8",
LC_NAME = "en_US.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
Pipeline Complete
Pipeline Complete
When I ran:
silviav@mutant9:/bin:bash ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_te_families.tsv -1 /hosts/linuxhome/chaperone/silviav/reads/Kita/SRR4209894_paired_1.fastq -2 /hosts/linuxhome/chaperone/silviav/reads/Kita/SRR4209894_paired_2.fastq -o /hosts/linuxhome/chaperone/silviav/reads/Kita/McClintock_results -p 24
works, but when I ran the python script in /bin/
as python /hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_Kita.py
indicating:
#!/usr/bin/env python
import os
import subprocess
if __name__=='__main__':
path = "/hosts/linuxhome/chaperone/silviav/reads/Gallone"
dir_files = os.listdir(path)
pair_reads = {}
for file in sorted(dir_files):
if file.endswith("_paired_R1.fastq"):
file1 = file
if file.endswith("_paired_R2.fastq"):
file2 = file
pair_reads[file1] = file2
for key, value in pair_reads.items():
code = "/bin/bash ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_te_families.tsv -1 {} -2 {} -o /hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results -p 36".format(key, value)
cmd = subprocess.check_output(code, shell =True)
print( "EXIT STATUS AND TYPE", cmd)
it doesn't work as I get:
Median insert size =
runpopoolationte.sh: line 41: * 3 + -1: syntax error: operand expected (error token is "* 3 + -1")
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
mv: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results/sacCer2/SRR5678544_paired_R1/PoPoolationTE/SRR5678544_paired_R1_popoolationte*': No such file or directory
cp: cannot stat '/hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results/sacCer2/SRR5678544_paired_R1/PoPoolationTE/te-poly-filtered.txt': No such file or directory
unzip: cannot find or open *.zip, *.zip.zip or *.zip.ZIP.
No zipfiles found.
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results/sacCer2/SRR5678544_paired_R1/results/summary/fastqc_analysis/*_1*/fastqc_data.txt': No such file or directory
ls: cannot access '/hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results/sacCer2/SRR5678544_paired_R1/results/summary/fastqc_analysis/*_2*/fastqc_data.txt': No such file or directory
And this is with another approach:
/hosts/linuxhome/chaperone/silviav/reads/Gallone:for i in *paired_R1.fastq ; do ~/mcclintock/mcclintock.sh -r ~/mcclintock/test/sacCer2.fasta -c ~/mcclintock/test/sac_cer_TE_seqs.fasta -g ~/mcclintock/test/reference_TE_locations.gff -t ~/mcclintock/test/sac_cer_families.tsv -1 $i -2 "${i:0:19}2.fastq" -o /hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results -p 36; done
Running McClintock version:
Running McClintock version: 1d06a94bf7bcd5a1e17dfc8336fb82895780c713
1d06a94bf7bcd5a1e17dfc8336fb82895780c713
Date of run is 24_02_20
Date of run is 24_02_20
Creating directory structure...
Creating directory structure...
cp: cannot stat 'SRR5678544_paired_R1.fastq': No such file or directory
cp: cannot stat 'SRR5678544_paired_R2.fastq': No such file or directory
cp: cannot stat '/home/silviav/mcclintock/test/sac_cer_families.tsv': No such file or directory
rm: remove regular file '/hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results/sacCer2/reference/popool_te_seqs.fasta'?
But those two files exist:
/hosts/linuxhome/chaperone/silviav/reads/Gallone:ls -lsrth SRR5678544_paired_R1.fastq
2.3G -rw-r--r-- 1 silviav binf 2.3G Dec 12 13:10 SRR5678544_paired_R1.fastq
/hosts/linuxhome/chaperone/silviav/reads/Gallone:ls -lsrth SRR5678544_paired_R2.fastq
2.3G -rw-r--r-- 1 silviav binf 2.3G Dec 12 13:10 SRR5678544_paired_R2.fastq
Additionally, miniconda is activated:
/hosts/linuxhome/chaperone/silviav/reads/Gallone:which activate
activate is /home/silviav/miniconda/bin/activate
I'd really apreciate your help. Thank you for your time. Silvia
os.listdir
, which only produces relative paths. The fact that mcclintock.sh
can not read the file path suggests that the relative paths are not consistent with your current working directory. You provide absolute path for all the input files except for fastq files, I suggest you provide absolute path for fastq as well.Thank you so much for your help. It worked. On the other hand, I had to add --force
in the remove commands so maybe for your future updates you can introduce this additional remark.
Thank you for your feedback. It is really gratifying to see that you give support to the users.
Cheers,
Silvia
One additional issue. It seems that the pipeline for one pair is done and to continue with the next one I have to either overwrite the previousfasta.old
file or not. Which one is recommended? and how can I run McClintock without having to literally answer to any question? I'm running hundred of files and having to answer every time there is a new file coming won't be handy.
Thank you in advance
2 archives were successfully processed.
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
Pipeline Complete
Pipeline Complete
Running McClintock version:
Running McClintock version: 1d06a94bf7bcd5a1e17dfc8336fb82895780c713
1d06a94bf7bcd5a1e17dfc8336fb82895780c713
Date of run is 26_02_20
Date of run is 26_02_20
Creating directory structure...
Creating directory structure...
cp: overwrite '/hosts/linuxhome/chaperone/silviav/reads/Gallone/McClintock_results/sacCer2/reference/sacCer2.fasta.old'?
fasta.old
would be overwrited without prompt.Thank you so much. It is working so far.
Hi! I'm facing errors or warnings that the respective files don't exist. My dataset has fastq.gz files, is it necessary to unzip the files? This is the stdout:
Thank you for your help