search

sdparekh / zUMIs

zUMIs: A fast and flexible pipeline to process RNA sequencing data with UMIs
GNU General Public License v3.0
275 stars 68 forks source link

zUMIs.YAML.error.log: All files "NULL" #330

Closed carmensandoval closed 2 years ago

carmensandoval commented 2 years ago

Hi,

I can't figure out what I'm doing wrong in setting up zUMIs. This is what my YAML looks like:

project: bulk_smartseq_umis
sequence_files:
  file1:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_R1_001.fastq.gz
    base_definition:
      - cDNA(23-150)
      - UMI(12-19)
    find_pattern: ATTGCGCAATG
  file2:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_R2_001.fastq.gz
    base_definition:
      - cDNA(1-150)
  file3:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_I1_001.fastq.gz
    base_definition:
      - BC(1-8)
  file4:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_I2_001.fastq.gz
    base_definition:
      - BC(1-8)
reference:
  STAR_index: /path/to/refdata-gex-GRCh38-2020-A/star/
  GTF_file: /path/to/refdata-gex-GRCh38-2020-A/genes/genes.gtf
  additional_STAR_params: '--clip3pAdapterSeq CTGTCTCTTATACACATCT'
out_dir: /path/to/bulk_RNA/zUMIs
num_threads: 20
mem_limit: 128
filter_cutoffs:
  BC_filter:
    num_bases: 3
    phred: 20
  UMI_filter:
    num_bases: 2
    phred: 20
barcodes:
  barcode_num: 18
  barcode_file: 
  automatic: yes
  BarcodeBinning: 1
  nReadsperCell: 50000000
  demultiplex: no
counting_opts:
  introns: yes
  downsampling: '0'
  strand: 0
  Ham_Dist: 1
  write_ham: no
  velocyto: no
  primaryHit: yes
  twoPass: no
make_stats: yes
which_Stage: Filtering
samtools_exec: samtools
pigz_exec: pigz
STAR_exec: STAR
Rscript_exec: Rscript
zUMIs_directory: /path/to/applications/zUMIs

Many fastqs are created in the merged.tmp directory, but there is no output at the end of the run, and I always see this in the error log:

cat zUMIs.YAML.error.log

$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

[1] ""
[1] ""
[1] ""
[1] ""
[1] "" "" "" ""
[1] ""
[1] ""
[1] ""
[1] ""
[1] "" "" "" ""
[1] "NULL" "NULL" "NULL" "NULL"
$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

[1]

Any idea what I could be doing wrong?

Thanks!

cziegenhain commented 2 years ago

Hi,

Can you share the full verbose of the zUMIs run?

Edit: From your YAML file you have an empty string in the barcode_file field. If you do not want to use expected barcode file, please set a barcode_file: ~ (YAML spec requires the tilda for missing value). Also, there is no implementation of automatic barcode detection + top N barcodes like in your yaml. The pipeline should just ignore the automatic detection part if I remember right, but if you really want the top 18 barcodes, simply set automatic: no

carmensandoval commented 2 years ago

Thanks so much for your response.

OK, so I added a file expected_barcodes.txt which contains the concatenated (8bp) dual indexes for each of my samples.

cat expected_barcodes.txt

CCGCGGTTAGCGCTAG
AGTTCAGGCCAACAGA
TAATACAGATATTCAC
ACTAAGATAACCGCGG
TACCGAGGCCTGAACT
ATATGGATCTGTATTA
ATATCTCGATCTTAGT
TGCGGCGTCCTCGGTA
ACACTAAGATCCATAT
CAATTAACCGAGATAT
TTATAACCGATATCGA
GACCTGAATTGGTGAG
CGGCGTGAGCGCCTGT
GTCGGAGCGGTTATAA
CGTTAGAATTCAGGTC
GCGCAAGCTCACGCCG
GCGCTCTAGCTCCGAC
CATAATACTTCTAACG

I modified the YAML to indicate that I have a list of 18 expected barcodes:

project: bulk_smartseq_umis
sequence_files:
  file1:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_R1_001.fastq.gz
    base_definition:
      - cDNA(23-52)
      - UMI(12-19)
    find_pattern: ATTGCGCAATG
  file2:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_R2_001.fastq.gz
    base_definition:
      - cDNA(1-70)
  file3:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_I1_001.fastq.gz
    base_definition:
      - BC(1-8)
  file4:
    name: /path/to/bulk_RNA/bcl2fastq/fastqs/Undetermined_S0_I2_001.fastq.gz
    base_definition:
      - BC(1-8)
reference:
  STAR_index: /path/to/refdata-gex-GRCh38-2020-A/star/
  GTF_file: /path/to/refdata-gex-GRCh38-2020-A/genes/genes.gtf
  additional_STAR_params: '--clip3pAdapterSeq CTGTCTCTTATACACATCT'
out_dir: /path/to/bulk_RNA/zUMIs
num_threads: 20
mem_limit: 128
filter_cutoffs:
  BC_filter:
    num_bases: 3
    phred: 20
  UMI_filter:
    num_bases: 2
    phred: 20
barcodes:
  barcode_num: 18
  barcode_file: expected_barcodes.txt
  automatic:np
  BarcodeBinning: 1
  nReadsperCell: 1000
  demultiplex: no
counting_opts:
  introns: yes
  downsampling: '0'
  strand: 0
  Ham_Dist: 1
  write_ham: no
  velocyto: no
  primaryHit: yes
  twoPass: no
make_stats: yes
which_Stage: Filtering
samtools_exec: samtools
pigz_exec: pigz
STAR_exec: STAR
Rscript_exec: Rscript
zUMIs_directory: /path/to/applications/zUMIs

However, I still get NULL for all files in the zUMIs_YAMLerror.log:

WARNING: ignoring environment value of R_HOME
$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

[1] ""
[1] ""
[1] ""
[1] ""
[1] "" "" "" ""
[1] ""
[1] ""
[1] ""
[1] ""
[1] "" "" "" ""
[1] "NULL" "NULL" "NULL" "NULL"
$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

$file1
NULL

$file2
NULL

$file3
NULL

$file4
NULL

[1] 0

job .out:

Using miniconda environment for zUMIs!
 note: internal executables will be used instead of those specified in the YAML file!

 You provided these parameters:
 YAML file: zUMI.yaml
 zUMIs directory:       /path/to/applications/zUMIs
 STAR executable        STAR
 samtools executable        samtools
 pigz executable        pigz
 Rscript executable     Rscript
 RAM limit:   128
 zUMIs version 2.9.7c 

Tue Sep 20 08:52:51 PDT 2022

Filtering...

WARNING: ignoring environment value of R_HOME
Mapping...
WARNING: ignoring environment value of R_HOME
[1] "2022-09-20 09:54:05 PDT"
Tue Sep 20 09:54:07 PDT 2022
Counting...
WARNING: ignoring environment value of R_HOME
[1] "2022-09-20 09:54:22 PDT"
Tue Sep 20 09:54:23 PDT 2022
WARNING: ignoring environment value of R_HOME
[1] "loomR found"
Tue Sep 20 09:54:26 PDT 2022
Descriptive statistics...
WARNING: ignoring environment value of R_HOME
[1] "I am loading useful packages for plotting..."
[1] "2022-09-20 09:54:26 PDT"
Tue Sep 20 09:54:44 PDT 2022

When the job finishes, there is no output in /expression or /stats . bulk_smartseq.filtered.tagged.Log.final.out and bulk_smartseq.BCstats.txt are both empty.

I also noticed that in zUMIs_output/.tmpMerge, while there are 18 Undetermined_S0_I1_001 fastqs, there are different barcodes in each. Is this expected behavior?

For example:

less zUMIs_output/.tmpMerge/Undetermined_S0_I1_001.fastqsomaseq_bulk_smartseqaa.gz.

@VH00865:3:AAATMTMHV:1:1101:18383:1000 1:N:0:0
GTCGGAGC
+
CCCCCCCC
@VH00865:3:AAATMTMHV:1:1101:18421:1000 1:N:0:0
ACTTAGAT
+
CCC-;C-C
@VH00865:3:AAATMTMHV:1:1101:18459:1000 1:N:0:0
TGCGGCGT
+
CCC;CCCC
@VH00865:3:AAATMTMHV:1:1101:18496:1000 1:N:0:0
AGTTCAGG
+
CCCCCCCC
@VH00865:3:AAATMTMHV:1:1101:18610:1000 1:N:0:0
ATATGGAT
+
CC-C;CCC

Thank you, Carmen

cziegenhain commented 2 years ago

Hi Carmen,Looks good!The temp folder files are just arbitrary chunks so that should be fine.The error log has only NULL that’s also ok as it ends in a 0 (no error). Sorry this file is a bit cryptic it was more meant to be an internal diagnostics tool. Anyway it seems like it is running now as it should, right?BestChristoph

carmensandoval commented 2 years ago

Sadly no, when the job finishes, there is no output in /expression or /stats . bulk_smartseq.filtered.tagged.Log.final.out and bulk_smartseq.BCstats.txt are both empty.

Here is what I see in the job's .err file:

couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqar.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqaq.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqal.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqaa.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqac.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqag.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqad.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqam.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqae.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqaf.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqan.filtered.tagged.bam'
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqah.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqao.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqaj.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqak.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqap.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqab.filtered.tagged.bam'
couldn't open WARNING: for reading: No such file or directory
couldn't open ignoring for reading: No such file or directory
couldn't open environment for reading: No such file or directory
couldn't open value for reading: No such file or directory
couldn't open of for reading: No such file or directory
couldn't open R_HOME for reading: No such file or directory
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `samtools view -Sb - > cDNA(23-52);UMI(12-19) cDNA(1-70) BC(1-8) BC(1-8)/zUMIs_output/.tmpMerge//gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes.somaseq_bulk_smartseqai.filtered.tagged.bam'
ls: cannot access /gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes/zUMIs_output/.tmpMerge//somaseq_bulk_smartseq.*.filtered.tagged.bam: No such file or directory
cat: /gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes/zUMIs_output/.tmpMerge//somaseq_bulk_smartseq.*.BCstats.txt: No such file or directory
[main_samview] fail to read the header from "-".
Error in eval(bysub, parent.frame(), parent.frame()) : 
  object 'XC' not found
Calls: cellBC -> [ -> [.data.table -> eval -> eval
In addition: Warning message:
In data.table::fread(bccount_file, header = FALSE, col.names = c("XC",  :
  File '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes/somaseq_bulk_smartseq.BCstats.txt' has size 0. Returning a NULL data.table.
Execution halted
sh: 1: samtools: not found
Warning message:
In data.table::fread(cmd = paste(samtools, "view", filtered_bams[1],  :
  File '/local/tmp/Rtmp1R9oWy/file165971b42dc1' has size 0. Returning a NULL data.table.
sh: 1: sh: 1: sh: 1: STAR: not found
sh: 1: STAR: not found
sh: 1: STAR: not found
sh: 1: STAR: not found
sh: 1: sh: 1: sh: 1: sh: 1: STAR: not found
samtools: not found
STAR: not found
STAR: not found
STAR: not found
STAR: not found
sh: 1: samtools: not found
sh: 1: samtools: not found
rm: cannot remove '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes/zUMIs_output/.tmpMap/tmp.somaseq_bulk_smartseq.*': No such file or directory
Error in fread(paste0(opt$out_dir, "/zUMIs_output/", opt$project, "kept_barcodes_binned.txt")) : 
  File '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes/zUMIs_output/somaseq_bulk_smartseqkept_barcodes_binned.txt' does not exist or is non-readable. getwd()=='/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes'
Execution halted
Loading required package: yaml
Loading required package: Matrix
Error in gzfile(file, "rb") : cannot open the connection
Calls: rds_to_loom -> readRDS -> gzfile
In addition: Warning message:
In gzfile(file, "rb") :
  cannot open compressed file '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes/zUMIs_output/expression/somaseq_bulk_smartseq.dgecounts.rds', probable reason 'No such file or directory'
Execution halted
Error in data.table::fread(paste0(opt$out_dir, "/zUMIs_output/", opt$project,  : 
  File '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes/zUMIs_output/somaseq_bulk_smartseqkept_barcodes.txt' does not exist or is non-readable. getwd()=='/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_with_barcodes'
Execution halted
carmensandoval commented 2 years ago

This seems to be related to the issue:

https://github.com/sdparekh/zUMIs/issues/184#issuecomment-629319589

It seems to be taking the stderr(?) of "WARNING: ignoring environment value of R_HOME" and piping it in as input somehow?

What's further strange is that earlier logs showed a more coherent (non-split) stream of warnings that weren't getting piped as input... 

Filtering...
Tue Sep 20 09:54:01 PDT 2022
WARNING: ignoring environment value of R_HOME
Mapping...
WARNING: ignoring environment value of R_HOME
[1] "2022-09-20 09:54:05 PDT"
Tue Sep 20 09:54:07 PDT 2022
Counting...
WARNING: ignoring environment value of R_HOME
[1] "2022-09-20 09:54:22 PDT"
Tue Sep 20 09:54:23 PDT 2022
WARNING: ignoring environment value of R_HOME
[1] "loomR found"
Tue Sep 20 09:54:26 PDT 2022
Descriptive statistics...
WARNING: ignoring environment value of R_HOME
[1] "I am loading useful packages for plotting..."
[1] "2022-09-20 09:54:26 PDT"
Tue Sep 20 09:54:44 PDT 2022
carmensandoval commented 2 years ago

Alright - so unsetting R_HOME seems to get rid of the warning being passed on as input...

I think the remaining issues are coming from the way the scripts call the required modules. I am on a cluster, and the modules for STAR, RScript and samtools need to be loaded before those tools are available. I am loading those modules in my slurm script, but somehow that's not enough for the subscripts to see them.

Do you have any advice on how to handle module loading for zUMIs?

ml purge

ml load star
ml load samtools
# ml load Anaconda3
ml load R

which STAR
which samtools
which Rscript

unset R_HOME

/gstore/home/sandovc9/applications/zUMIs/zUMIs.sh -y zUMIs_tiny.yaml

job_id.out

/gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR
/gstore/apps/samtools/1.14-foss-2017a/bin/samtools
/gstore/apps/modulefiles/RP/bin/Rscript

 You provided these parameters:
 YAML file: zUMIs_tiny.yaml
 zUMIs directory:       /gstore/home/sandovc9/applications/zUMIs
 STAR executable        /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR
 samtools executable        /gstore/apps/samtools/1.14-foss-2017a/bin/samtools
 pigz executable        pigz
 Rscript executable     /gstore/apps/modulefiles/RP/bin/Rscript
 RAM limit:   256
 zUMIs version 2.9.7c 

Tue Sep 20 21:36:31 PDT 2022
WARNING: The STAR version used for mapping is 2.7.9a and the STAR index was created using the version 20201. This may lead to an error while mapping. If you encounter any errors at the mapping stage, please make sure to create the STAR index using STAR 2.7.9a.
Filtering...
Tue Sep 20 21:36:36 PDT 2022
Mapping...
[1] "2022-09-20 21:36:45 PDT"
Tue Sep 20 21:36:45 PDT 2022
Counting...
Tue Sep 20 21:36:56 PDT 2022
[1] "loomR found"
Tue Sep 20 21:37:24 PDT 2022
Descriptive statistics...
[1] "I am loading useful packages for plotting..."
[1] "2022-09-20 21:37:27 PDT"
Tue Sep 20 21:37:38 PDT 2022

job_id.err

You provided these parameters:
 YAML file:     zUMIs_tiny.yaml
 zUMIs directory:               /gstore/home/sandovc9/applications/zUMIs
 STAR executable                /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR
 samtools executable            /gstore/apps/samtools/1.14-foss-2017a/bin/samtools
 pigz executable                pigz
 Rscript executable             /gstore/apps/modulefiles/RP/bin/Rscript
 RAM limit:   256
 zUMIs version 2.9.7c 

Tue Sep 20 21:37:31 PDT 2022

Filtering...

sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file

Tue Sep 20 21:37:35 PDT 2022
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '/barcodeIDFUN.R': No such file or directory
Execution halted
Mapping...
[1] "2022-09-20 21:37:45 PDT"
cp: cannot create regular file '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_tiny/somaseq_bulk_smartseq_tiny.final_annot.gtf': Permission denied
sh: 1: /gstore/apps/samtools/1.14-foss-2017a/bin/samtools: not found
Warning message:
In data.table::fread(cmd = paste(samtools, "view", filtered_bams[1],  :
  File '/local/tmp/RtmpGeoZVT/file1b9568d4ed47' has size 0. Returning a NULL data.table.
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found/gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found

sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/star/2.7.9a-GCCcore-6.3.0/bin/STAR: not found
sh: 1: /gstore/apps/samtools/1.14-foss-2017a/bin/samtools: not found
sh: 1: sh: 1: /gstore/apps/samtools/1.14-foss-2017a/bin/samtools: not found/gstore/apps/samtools/1.14-foss-2017a/bin/samtools: not found

rm: cannot remove '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_tiny/zUMIs_output/.tmpMap/tmp.somaseq_bulk_smartseq_tiny.*': No such file or directory
Tue Sep 20 21:37:45 PDT 2022
Counting...
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '/runfeatureCountFUN.R': No such file or directory
Execution halted
Tue Sep 20 21:37:58 PDT 2022
Loading required package: yaml
Loading required package: Matrix

[1] "loomR found"
Error in gzfile(file, "rb") : cannot open the connection
Calls: rds_to_loom -> readRDS -> gzfile
In addition: Warning message:
In gzfile(file, "rb") :
  cannot open compressed file '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_tiny/zUMIs_output/expression/somaseq_bulk_smartseq_tiny.dgecounts.rds', probable reason 'No such file or directory'
Execution halted
Tue Sep 20 21:38:27 PDT 2022
Descriptive statistics...
[1] "I am loading useful packages for plotting..."
[1] "2022-09-20 21:38:29 PDT"
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '/statsFUN.R': No such file or directory
Execution halted
carmensandoval commented 2 years ago

So I switched back to using the conda option, and I no longer run into the STAR/samtools not found errors, but I still see a bunch of i/o errors and a cryptic unexpected EOF while looking for matching''`

sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '/barcodeIDFUN.R': No such file or directory
Execution halted
cp: cannot create regular file ‘/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_tiny/somaseq_bulk_smartseq_tiny.final_annot.gtf’: Permission denied
Warning message:
In data.table::fread(cmd = paste(samtools, "view", filtered_bams[1],  :
  File '/local/51222332/RtmpPDWz6B/file556d2fb0f74a' has size 0. Returning a NULL data.table.
[E::hts_open_format] Failed to open file NA
samtools view: failed to open "NA" for reading: No such file or directory
[E::hts_open_format] Failed to open file NA
samtools view: failed to open "NA" for reading: No such file or directory

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting
EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting
EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting

EXITING because of fatal PARAMETERS error: pGe.sjdbOverhang <=0 while junctions are inserted on the fly with --sjdbFileChrStartEnd or/and --sjdbGTFfile
SOLUTION: specify pGe.sjdbOverhang>0, ideally readmateLength-1
Sep 20 22:47:40 ...... FATAL ERROR, exiting
[main_cat] ERROR: input is not BAM or CRAM
[main_cat] ERROR: input is not BAM or CRAM
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '/runfeatureCountFUN.R': No such file or directory
Execution halted
Loading required package: yaml
Loading required package: Matrix
Error in gzfile(file, "rb") : cannot open the connection
Calls: rds_to_loom -> readRDS -> gzfile
In addition: Warning message:
In gzfile(file, "rb") :
  cannot open compressed file '/gstore/data/ctgbioinfo/sandovc9/soma-seq/bulk_RNA/zUMIs_tiny/zUMIs_output/expression/somaseq_bulk_smartseq_tiny.dgecounts.rds', probable reason 'No such file or directory'
Execution halted
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file '/statsFUN.R': No such file or directory
Execution halted
cziegenhain commented 2 years ago

Hi,

This seems all to be related to path issues from running within your SLURM/HPC environment. Sadly that is something I can't test on my side.

From the other issue you opened it seems like you managed to get the test dataset running, is it ok to close this issue then?

carmensandoval commented 2 years ago

Hi @cziegenhain -

Indeed, there were a bunch of issues related to running this on the HPC.

The conda version seems to be working with the test dataset, as well as with a tiny subset of my data, but I am seeing an issue where larger files (the full run) fail to produce the full output (in particular the .rds files).

Will open a new issue.

Appreciate your help!