PierreBSC / Viral-Track

MIT License
54 stars 27 forks source link

Doubts processing COVID-19 samples #12

Open vanessassilva opened 3 years ago

vanessassilva commented 3 years ago

Hi Pierre Bost,

I downloaded the nine FASTQ files for series GSE145926, via SRA in the original format, conform reported from paper. So I end up with two files the each sample. Did you use the 18 FASTQ files of COVID-19 samples to analysis? if so, after doing all the preparation of the files, how long did it take to run the scripts? When I run Viral-Track Scanning.R script, reported me the error:

samtools index: "/home/vanessa/star_cv/output_viral_track//SRR11181955_2//SRR11181955_2Aligned.sortedByCoord.out.bam" is in a format that cannot be usefully indexed Indexing of the bam file for SRR11181955_2 is done samtools idxstats: fail to load index for "/home/vanessa/star_cv/output_viral_track//SRR11181955_2//SRR11181955_2Aligned.sortedByCoord.out.bam" Computing stat file for the bam file for SRR11181955_2 is done Error in read.table(temp_chromosome_count_path, header = F, row.names = 1) : no lines available in input Execution halted

What am I doing wrong? What solution do you suggest?

Another question, about the analysis with MetaCell, this analysis was carried out separated of the scripts, correct?

Best regards, Vanessa

yangjywhu commented 3 years ago

I have the same problem:

[Mon Aug 31 19:42:08 2020]
rule viraltrack:
    input: result/viraltrack/umi_tools/p1_R2_extraced.fastq.gz, ref/hg38_WIV04/STARindex/STARindex, script/Viral-Track/Viral_Track_scanning.R, script/Viral-Track/Virusite_annotation_file.txt
    output: result/viraltrack/scanning/p1_R2/Parameters.txt, result/viraltrack/scanning/p1_R2/Files_to_process.txt, result/viraltrack/scanning/p1_R2/outs
    jobid: 1
    wildcards: sample=p1, read=R2

Loading of the libraries.... ... done ! Warning message:
Output directory does not exist ! Creating it ! 
1 Fastq files are going to be processed ! 
Mapping p1_R2_extraced.fastq file 
Aug 31 19:42:40 ..... started STAR run
Aug 31 19:42:40 ..... loading genome
Aug 31 20:19:08 ..... started 1st pass mapping
Mapping ofp1_R2_extraced.fastq done ! 
All fastq files have been mapped successfully 
Starting the BAM file analysis 
samtools index: "result/viraltrack/scanning/p1_R2/outs/p1_R2_extraced//p1_R2_extraced_Aligned.sortedByCoord.out.bam" is in a format that cannot be usefully indexed
Indexing of the bam file for p1_R2_extraced is done 
samtools idxstats: fail to load index for "result/viraltrack/scanning/p1_R2/outs/p1_R2_extraced//p1_R2_extraced_Aligned.sortedByCoord.out.bam"
Computing stat file for the bam file for p1_R2_extraced is done 
Error in read.table(temp_chromosome_count_path, header = F, row.names = 1) : 
  no lines available in input
Execution halted
[Mon Aug 31 20:19:37 2020]
Error in rule viraltrack:
    jobid: 1
    output: result/viraltrack/scanning/p1_R2/Parameters.txt, result/viraltrack/scanning/p1_R2/Files_to_process.txt, result/viraltrack/scanning/p1_R2/outs
    shell:

        echo 'N_thread=1
Output_directory="result/viraltrack/scanning/p1_R2/outs"
Index_genome="ref/hg38_WIV04/STARindex/STARindex"
Viral_annotation_file="script/Viral-Track/Virusite_annotation_file.txt"
Name_run="cov2sc_p1_R2"
Load_STAR_module=FALSE
Load_samtools_module=FALSE
Load_stringtie_module=FALSE
Minimal_read_mapped=5
Single_cell_metadata="script/config.yaml"
' > result/viraltrack/scanning/p1_R2/Parameters.txt
        echo "result/viraltrack/umi_tools/p1_R2_extraced.fastq.gz" > result/viraltrack/scanning/p1_R2/Files_to_process.txt
        set +u; source ~/miniconda3/bin/activate viraltrack; set -u
        Rscript script/Viral-Track/Viral_Track_scanning.R result/viraltrack/scanning/p1_R2/Parameters.txt result/viraltrack/scanning/p1_R2/Files_to_process.txt
        set +u; conda deactivate; set -u

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job viraltrack since they might be corrupted:
result/viraltrack/scanning/p1_R2/Parameters.txt, result/viraltrack/scanning/p1_R2/Files_to_process.txt, result/viraltrack/scanning/p1_R2/outs
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
PierreBSC commented 3 years ago

Dear Vanessa and Jiayi,

Thank you for your feedback. This is indeed weird : first according to Jiayi's log file, STAR only performs a single-pass mapping and not a two-pass mapping, but this should only be secondary and not causes this problem. Can you check that the resulting bam exists and is not empty ? Best

Pierre

yangjywhu commented 3 years ago

Hello @PierreBSC,

The output directory is:

yangjiayi@localhost:~/project/cov2sc/result/viraltrack/scanning/p1_R2/outs/p1_R2_extraced
$ ll
total 52K
-rw-r--r--. 1 yangjiayi zhoulab    0 Sep  1 00:25 Count_chromosomes.txt
-rw-r--r--. 1 yangjiayi zhoulab    0 Aug 31 23:16 p1_R2_extraced_Aligned.sortedByCoord.out.bam
-rw-r--r--. 1 yangjiayi zhoulab  29K Sep  1 00:25 p1_R2_extraced_Log.out
-rw-r--r--. 1 yangjiayi zhoulab 5.1K Sep  1 00:25 p1_R2_extraced_Log.progress.out
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:22 p1_R2_extraced__STARgenome
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:22 p1_R2_extraced__STARpass1
drwx------. 3 yangjiayi zhoulab 4.0K Sep  1 00:24 p1_R2_extraced__STARtmp

Unfortunately, the bam file is empty, but Log.progress.out is not empty, and it tell me Finished 1st pass mapping But, the bug is reported in Log.out:

BAMoutput.cpp:27:BAMoutput: exiting because of *OUTPUT FILE* error: could not create output file result/viraltrack/scanning/p1_R2/outs/p1_R2_extraced/p1_R2_extraced__STARtmp//BAMsort/19/46
SOLUTION: check that the path exists and you have write permission for this file. Also check ulimit -n and increase it to allow more open files.

So i check p1_R2_extraced__STARtmp//BAMsort.

yangjiayi@localhost:~/project/cov2sc/result/viraltrack/scanning/p1_R2/outs/p1_R2_extraced
$ ll p1_R2_extraced__STARtmp
total 5.2M
drwx------. 22 yangjiayi zhoulab 4.0K Sep  1 00:25 BAMsort
-rw-r--r--.  1 yangjiayi zhoulab  109 Sep  1 00:25 readFilesIn.info
-rwx------.  1 yangjiayi zhoulab  177 Sep  1 00:25 readsCommand_read1
-rw-r--r--.  1 yangjiayi zhoulab 5.2M Sep  1 00:22 SJ.start_gap.tsv
prw-------.  1 yangjiayi zhoulab    0 Sep  1 00:25 tmp.fifo.read1
yangjiayi@localhost:~/project/cov2sc/result/viraltrack/scanning/p1_R2/outs/p1_R2_extraced
$ ll p1_R2_extraced__STARtmp/BAMsort/
total 80K
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 0
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 1
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 10
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 11
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 12
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 13
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 14
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 15
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 16
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 17
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 18
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 19
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 2
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 3
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 4
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 5
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 6
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 7
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 8
drwx------. 2 yangjiayi zhoulab 4.0K Sep  1 00:25 9
yangjiayi@localhost:~/project/cov2sc/result/viraltrack/scanning/p1_R2/outs/p1_R2_extraced
$ ll p1_R2_extraced__STARtmp/BAMsort/19
total 0
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 0
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 1
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 10
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 11
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 12
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 13
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 14
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 15
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 16
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 17
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 18
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 19
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 2
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 20
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 21
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 22
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 23
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 24
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 25
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 26
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 27
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 28
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 29
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 3
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 30
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 31
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 32
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 33
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 34
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 35
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 36
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 37
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 38
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 39
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 4
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 40
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 41
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 42
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 43
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 44
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 45
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 5
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 6
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 7
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 8
-rw-r--r--. 1 yangjiayi zhoulab 0 Sep  1 00:25 9

Working directory is ~/project/cov2sc and i have permission on it. Thank you for your attention on this problem

Jiayi Yang

yangjywhu commented 3 years ago

hi @PierreBSC This problem still occurs, and I can’t use Viral-Track. Could you give me some suggestions? Output Directory is update above.

Thanks Jiayi