search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
991 stars 354 forks source link

scRNA-seq pipeline rapmap error #2696

Closed zji90 closed 5 years ago

zji90 commented 5 years ago

Hi, I am running scRNA-seq pipeline with rapmap and I get the following error:

/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/sparse/sputils.py:115: UserWarning: object dtype is not supported by sparse matrices warnings.warn("object dtype is not supported by sparse matrices") Traceback (most recent call last): File "/scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis", line 11, in load_entry_point('umis==1.0.3', 'console_scripts', 'umis')() File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/umis/umis.py", line 952, in sparse scipy.io.mmwrite(out_handle, scipy.sparse.csr_matrix(df)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 102, in mmwrite MMFile().write(target, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 451, in write self._write(stream, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 709, in _write raise TypeError('unexpected dtype kind ' + kind) TypeError: unexpected dtype kind O ' returned non-zero exit status 1

roryk commented 5 years ago

I'm thinking that there might be something off with your run-- could you post the log files and your configuration files? My guess is that there is something leading to zero counts being produced which is causing this error, but I'd need to look at the log files to figure it out.

zji90 commented 5 years ago

Sure. See below for the configuration and error message. I am running this on a computing cluster which uses centos system. I am specifying my own gtf file instead of the default one. I guess probably that's what went wrong? Thanks for your help!

yaml configuration file: upload: dir: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/align/run2 details:

Log: Note that users no longer need to load the SLURM module. You may remove 'module load slurm' from your scripts. Please read this resource guide to understand the above behavior: https://www.marcc.jhu.edu/getting-started/working-with-modules-on-centos-7/

Lmod is automatically replacing "intel/18.0" with "gcc/5.5.0".

Due to MODULEPATH changes, the following have been reloaded: 1) openmpi/3.1

Note that users no longer need to load the SLURM module. You may remove 'module load slurm' from your scripts. Please read this resource guide to understand the above behavior: https://www.marcc.jhu.edu/getting-started/working-with-modules-on-centos-7/ [2019-02-24T12:29Z] System YAML configuration: /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/bcbio_system.yaml [2019-02-24T12:29Z] Resource requests: picard; memory: 3.00; cores: 16 [2019-02-24T12:29Z] Configuring 6 jobs to run, using 1 cores each with 3.00g of memory reserved for each job [2019-02-24T12:29Z] Timing: organize samples [2019-02-24T12:29Z] multiprocessing: organize_samples [2019-02-24T12:29Z] Using input YAML configuration: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/run2.yaml [2019-02-24T12:29Z] Checking sample YAML configuration: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/run2.yaml [2019-02-24T12:29Z] Testing minimum versions of installed programs [2019-02-24T12:29Z] multiprocessing: prepare_sample [2019-02-24T12:29Z] Preparing E14-2i-D1 [2019-02-24T12:29Z] Preparing E14-SL-cond-2 [2019-02-24T12:29Z] Preparing E14-2i-D3 [2019-02-24T12:29Z] Preparing E14-2i-cond-2 [2019-02-24T12:29Z] Preparing E14-2i-D5 [2019-02-24T12:29Z] Preparing E14-2i-D7 [2019-02-24T12:29Z] Resource requests: rapmap, samtools; memory: 3.00, 3.00; cores: 16, 16 [2019-02-24T12:29Z] Configuring 1 jobs to run, using 16 cores each with 48.1g of memory reserved for each job [2019-02-24T12:29Z] Timing: singlecell-rnaseq [2019-02-24T12:29Z] multiprocessing: run_umi_transform [2019-02-24T12:29Z] multiprocessing: demultiplex_samples [2019-02-24T12:29Z] multiprocessing: run_filter_barcodes [2019-02-24T12:29Z] multiprocessing: run_barcode_histogram [2019-02-24T12:29Z] multiprocessing: run_rapmap_index [2019-02-24T12:29Z] multiprocessing: run_rapmap_align [2019-02-24T12:29Z] multiprocessing: run_tagcount [2019-02-24T12:29Z] Counting alignments of transcripts in /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D1-ATTAGACG/E14-2i-D1-ATTAGACG.bam. [2019-02-24T12:29Z] INFO:umis.umis:Reading optional files [2019-02-24T12:29Z] INFO:umis.umis:Keeping 3630 out of 147112 cellular barcodes. [2019-02-24T12:29Z] INFO:umis.umis:Tallying evidence [2019-02-24T12:32Z] Converting /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full to sparse format. [2019-02-24T12:33Z] /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/sparse/sputils.py:115: UserWarning: object dtype is not supported by sparse matrices [2019-02-24T12:33Z] warnings.warn("object dtype is not supported by sparse matrices") [2019-02-24T12:33Z] Traceback (most recent call last): [2019-02-24T12:33Z] File "/scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis", line 11, in [2019-02-24T12:33Z] load_entry_point('umis==1.0.3', 'console_scripts', 'umis')() [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 764, in call [2019-02-24T12:33Z] return self.main(args, kwargs) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 717, in main [2019-02-24T12:33Z] rv = self.invoke(ctx) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 1137, in invoke [2019-02-24T12:33Z] return _process_result(sub_ctx.command.invoke(sub_ctx)) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 956, in invoke [2019-02-24T12:33Z] return ctx.invoke(self.callback, ctx.params) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 555, in invoke [2019-02-24T12:33Z] return callback(args, kwargs) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/umis/umis.py", line 952, in sparse [2019-02-24T12:33Z] scipy.io.mmwrite(out_handle, scipy.sparse.csr_matrix(df)) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 102, in mmwrite [2019-02-24T12:33Z] MMFile().write(target, a, comment, field, precision, symmetry) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 451, in write [2019-02-24T12:33Z] self._write(stream, a, comment, field, precision, symmetry) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 709, in _write [2019-02-24T12:33Z] raise TypeError('unexpected dtype kind ' + kind) [2019-02-24T12:33Z] TypeError: unexpected dtype kind O [2019-02-24T12:33Z] Uncaught exception occurred Traceback (most recent call last): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 26, in run _do_run(cmd, checks, log_stdout, env=env) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 106, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) CalledProcessError: Command 'set -o pipefail; /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/sparse/sputils.py:115: UserWarning: object dtype is not supported by sparse matrices warnings.warn("object dtype is not supported by sparse matrices") Traceback (most recent call last): File "/scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis", line 11, in load_entry_point('umis==1.0.3', 'console_scripts', 'umis')() File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 555, in invoke return callback(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/umis/umis.py", line 952, in sparse scipy.io.mmwrite(out_handle, scipy.sparse.csr_matrix(df)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 102, in mmwrite MMFile().write(target, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 451, in write self._write(stream, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 709, in _write raise TypeError('unexpected dtype kind ' + kind) TypeError: unexpected dtype kind O ' returned non-zero exit status 1 Traceback (most recent call last): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/tool/bin/bcbio_nextgen.py", line 238, in main(kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/tool/bin/bcbio_nextgen.py", line 46, in main run_main(kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 45, in run_main fc_dir, run_info_yaml) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 89, in _run_toplevel for xs in pipeline(config, run_info_yaml, parallel, dirs, samples): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 304, in singlecellrnaseqpipeline samples = rnaseq.singlecell_rnaseq(samples, run_parallel) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/rnaseq.py", line 35, in singlecell_rnaseq samples = run_parallel("run_tagcount", samples) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel return run_multicore(fn, items, config, parallel=parallel) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(x) for x in items): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 921, in call if self.dispatch_one_batch(iterator): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch self._dispatch(tasks) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 716, in _dispatch job = self._backend.apply_async(batch, callback=cb) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/_parallel_backends.py", line 182, in apply_async result = ImmediateResult(func) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/_parallel_backends.py", line 549, in init self.results = batch() File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 225, in call for func, args, kwargs in self.items] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 54, in wrapper return f(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 27, in run_tagcount return umi.tagcount(*args) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/umi.py", line 285, in tagcount do.run(cmd.format(*locals()), message) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 26, in run _do_run(cmd, checks, log_stdout, env=env) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 106, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) subprocess.CalledProcessError: Command 'set -o pipefail; /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/sparse/sputils.py:115: UserWarning: object dtype is not supported by sparse matrices warnings.warn("object dtype is not supported by sparse matrices") Traceback (most recent call last): File "/scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis", line 11, in load_entry_point('umis==1.0.3', 'console_scripts', 'umis')() File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, *ctx.params) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/umis/umis.py", line 952, in sparse scipy.io.mmwrite(out_handle, scipy.sparse.csr_matrix(df)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 102, in mmwrite MMFile().write(target, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 451, in write self._write(stream, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 709, in _write raise TypeError('unexpected dtype kind ' + kind) TypeError: unexpected dtype kind O ' returned non-zero exit status 1

lpantano commented 5 years ago

Hi,

as well, can you take a look at the first line of the fasta used to do the index? I think you should have that inside the rapmap folder and then index folder.

I got a similar error when the fasta used was malformed but I don’t know if this is the case.

cheers

sent not from my computer

On Feb 24, 2019, at 20:32, Zhicheng Ji notifications@github.com wrote:

Sure. See below for the configuration and error message. I am running this on a computing cluster which uses centos system. I am specifying my own gtf file instead of the default one. I guess probably that's what went wrong? Thanks for your help!

yaml configuration file: upload: dir: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/align/run2 details:

analysis: scRNA-seq algorithm: umi_type: harvard-indrop-v3 singlecell_quantifier: rapmap minimum_barcode_depth: 1000 sample_barcodes: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-1.txt transcriptome_gtf: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation.gtf description: E14-2i-D1 files: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz genome_build: mm10 analysis: scRNA-seq algorithm: umi_type: harvard-indrop-v3 singlecell_quantifier: rapmap minimum_barcode_depth: 1000 sample_barcodes: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-2.txt transcriptome_gtf: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation.gtf description: E14-SL-cond-2 files: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz genome_build: mm10 analysis: scRNA-seq algorithm: umi_type: harvard-indrop-v3 singlecell_quantifier: rapmap minimum_barcode_depth: 1000 sample_barcodes: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-3.txt transcriptome_gtf: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation.gtf description: E14-2i-D3 files: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz genome_build: mm10 analysis: scRNA-seq algorithm: umi_type: harvard-indrop-v3 singlecell_quantifier: rapmap minimum_barcode_depth: 1000 sample_barcodes: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-4.txt transcriptome_gtf: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation.gtf description: E14-2i-cond-2 files: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz genome_build: mm10 analysis: scRNA-seq algorithm: umi_type: harvard-indrop-v3 singlecell_quantifier: rapmap minimum_barcode_depth: 1000 sample_barcodes: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-5.txt transcriptome_gtf: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation.gtf description: E14-2i-D5 files: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz genome_build: mm10 analysis: scRNA-seq algorithm: umi_type: harvard-indrop-v3 singlecell_quantifier: rapmap minimum_barcode_depth: 1000 sample_barcodes: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-6.txt transcriptome_gtf: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation.gtf description: E14-2i-D7 files: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz genome_build: mm10 Log: Note that users no longer need to load the SLURM module. You may remove 'module load slurm' from your scripts. Please read this resource guide to understand the above behavior: https://www.marcc.jhu.edu/getting-started/working-with-modules-on-centos-7/

Lmod is automatically replacing "intel/18.0" with "gcc/5.5.0".

Due to MODULEPATH changes, the following have been reloaded:

openmpi/3.1 Note that users no longer need to load the SLURM module. You may remove 'module load slurm' from your scripts. Please read this resource guide to understand the above behavior: https://www.marcc.jhu.edu/getting-started/working-with-modules-on-centos-7/ [2019-02-24T12:29Z] System YAML configuration: /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/bcbio_system.yaml [2019-02-24T12:29Z] Resource requests: picard; memory: 3.00; cores: 16 [2019-02-24T12:29Z] Configuring 6 jobs to run, using 1 cores each with 3.00g of memory reserved for each job [2019-02-24T12:29Z] Timing: organize samples [2019-02-24T12:29Z] multiprocessing: organize_samples [2019-02-24T12:29Z] Using input YAML configuration: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/run2.yaml [2019-02-24T12:29Z] Checking sample YAML configuration: /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/run2.yaml [2019-02-24T12:29Z] Testing minimum versions of installed programs [2019-02-24T12:29Z] multiprocessing: prepare_sample [2019-02-24T12:29Z] Preparing E14-2i-D1 [2019-02-24T12:29Z] Preparing E14-SL-cond-2 [2019-02-24T12:29Z] Preparing E14-2i-D3 [2019-02-24T12:29Z] Preparing E14-2i-cond-2 [2019-02-24T12:29Z] Preparing E14-2i-D5 [2019-02-24T12:29Z] Preparing E14-2i-D7 [2019-02-24T12:29Z] Resource requests: rapmap, samtools; memory: 3.00, 3.00; cores: 16, 16 [2019-02-24T12:29Z] Configuring 1 jobs to run, using 16 cores each with 48.1g of memory reserved for each job [2019-02-24T12:29Z] Timing: singlecell-rnaseq [2019-02-24T12:29Z] multiprocessing: run_umi_transform [2019-02-24T12:29Z] multiprocessing: demultiplex_samples [2019-02-24T12:29Z] multiprocessing: run_filter_barcodes [2019-02-24T12:29Z] multiprocessing: run_barcode_histogram [2019-02-24T12:29Z] multiprocessing: run_rapmap_index [2019-02-24T12:29Z] multiprocessing: run_rapmap_align [2019-02-24T12:29Z] multiprocessing: run_tagcount [2019-02-24T12:29Z] Counting alignments of transcripts in /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D1-ATTAGACG/E14-2i-D1-ATTAGACG.bam. [2019-02-24T12:29Z] INFO:umis.umis:Reading optional files [2019-02-24T12:29Z] INFO:umis.umis:Keeping 3630 out of 147112 cellular barcodes. [2019-02-24T12:29Z] INFO:umis.umis:Tallying evidence [2019-02-24T12:32Z] Converting /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full to sparse format. [2019-02-24T12:33Z] /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/sparse/sputils.py:115: UserWarning: object dtype is not supported by sparse matrices [2019-02-24T12:33Z] warnings.warn("object dtype is not supported by sparse matrices") [2019-02-24T12:33Z] Traceback (most recent call last): [2019-02-24T12:33Z] File "/scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis", line 11, in [2019-02-24T12:33Z] load_entry_point('umis==1.0.3', 'console_scripts', 'umis')() [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 764, in call [2019-02-24T12:33Z] return self.main(args, kwargs) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 717, in main [2019-02-24T12:33Z] rv = self.invoke(ctx) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 1137, in invoke [2019-02-24T12:33Z] return _process_result(sub_ctx.command.invoke(sub_ctx)) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 956, in invoke [2019-02-24T12:33Z] return ctx.invoke(self.callback, ctx.params) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 555, in invoke [2019-02-24T12:33Z] return callback(args, kwargs) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/umis/umis.py", line 952, in sparse [2019-02-24T12:33Z] scipy.io.mmwrite(out_handle, scipy.sparse.csr_matrix(df)) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 102, in mmwrite [2019-02-24T12:33Z] MMFile().write(target, a, comment, field, precision, symmetry) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 451, in write [2019-02-24T12:33Z] self._write(stream, a, comment, field, precision, symmetry) [2019-02-24T12:33Z] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 709, in _write [2019-02-24T12:33Z] raise TypeError('unexpected dtype kind ' + kind) [2019-02-24T12:33Z] TypeError: unexpected dtype kind O [2019-02-24T12:33Z] Uncaught exception occurred Traceback (most recent call last): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 26, in run _do_run(cmd, checks, log_stdout, env=env) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 106, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) CalledProcessError: Command 'set -o pipefail; /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/sparse/sputils.py:115: UserWarning: object dtype is not supported by sparse matrices warnings.warn("object dtype is not supported by sparse matrices") Traceback (most recent call last): File "/scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis", line 11, in load_entry_point('umis==1.0.3', 'console_scripts', 'umis')() File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 555, in invoke return callback(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/umis/umis.py", line 952, in sparse scipy.io.mmwrite(out_handle, scipy.sparse.csr_matrix(df)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 102, in mmwrite MMFile().write(target, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 451, in write self._write(stream, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 709, in _write raise TypeError('unexpected dtype kind ' + kind) TypeError: unexpected dtype kind O ' returned non-zero exit status 1 Traceback (most recent call last): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/tool/bin/bcbio_nextgen.py", line 238, in main(kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/tool/bin/bcbio_nextgen.py", line 46, in main run_main(kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 45, in run_main fc_dir, run_info_yaml) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 89, in _run_toplevel for xs in pipeline(config, run_info_yaml, parallel, dirs, samples): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 304, in singlecellrnaseqpipeline samples = rnaseq.singlecell_rnaseq(samples, run_parallel) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/rnaseq.py", line 35, in singlecell_rnaseq samples = run_parallel("run_tagcount", samples) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel return run_multicore(fn, items, config, parallel=parallel) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore for data in joblib.Parallel(parallel["num_jobs"], batch_size=1, backend="multiprocessing")(joblib.delayed(fn)(x) for x in items): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 921, in call if self.dispatch_one_batch(iterator): File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch self._dispatch(tasks) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 716, in _dispatch job = self._backend.apply_async(batch, callback=cb) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/_parallel_backends.py", line 182, in apply_async result = ImmediateResult(func) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/_parallel_backends.py", line 549, in init self.results = batch() File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 225, in call for func, args, kwargs in self.items] File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 54, in wrapper return f(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 27, in run_tagcount return umi.tagcount(*args) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/umi.py", line 285, in tagcount do.run(cmd.format(*locals()), message) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 26, in run _do_run(cmd, checks, log_stdout, env=env) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 106, in _do_run raise subprocess.CalledProcessError(exitcode, error_msg) subprocess.CalledProcessError: Command 'set -o pipefail; /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/sparse/sputils.py:115: UserWarning: object dtype is not supported by sparse matrices warnings.warn("object dtype is not supported by sparse matrices") Traceback (most recent call last): File "/scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis", line 11, in load_entry_point('umis==1.0.3', 'console_scripts', 'umis')() File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, *ctx.params) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/umis/umis.py", line 952, in sparse scipy.io.mmwrite(out_handle, scipy.sparse.csr_matrix(df)) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 102, in mmwrite MMFile().write(target, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 451, in write self._write(stream, a, comment, field, precision, symmetry) File "/home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/scipy/io/mmio.py", line 709, in _write raise TypeError('unexpected dtype kind ' + kind) TypeError: unexpected dtype kind O ' returned non-zero exit status 1

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

zji90 commented 5 years ago

Sure. Please see below. The file is under rapmap/index/quasiindex/mm10/mm10.fa It seems that there is an extra space between the gene name and the actual sequence. I don't know whether that will cause problem.

ENSMUST00000193812

aaggaaagaggataacacttgaaatgtaaataaagaaaatacctaataaaaataaataaaaacatgcttt caaaggaaataaaaagttggattcaaaaatttaacttttgctcatttggtataatcaaggaaaagacctt tgcatataaaatatattttgaataaaattcagtggaagaatggaatagaaatataagtttaaTGCTAAGT ATAAGTACCAGTAAAAGAATAATAAAAAGAAATATAAGTTGGGTATACAGTTATTTGCCAGCACAAAGCC TTGGGTATGGTTCTTAGCACTAAGGAACCAGCCAAATCACCAACAAACAGAGGCATAAGGTTTTAGTGTT TACTATTTGTACTTTTGTGGATCATCTTGCCAGCCTGTAGTGCAACCATCTCTAATCCACCACCATGAAG GGAACTGTGATAATTCACTGGGCTTTTTCTGTGCAAGATGAAAAAAAGCCAGGTGAGGCTGATTTATGAG TAAGGGATGTGCATTCCTAACTCAAAAATCTGAAATTTGAAATGCCGCCCAAAATATGAAATTTCACTGG CTGAGACGGTGTAATAGTCAGAAATTTCCATCTGTGATTGCTTGTAAAGTCAAATGTTAGAGACTATGTA TGTAGCATGTGCATGATATATACATGAAGTTGTATTCAGACTTGGGCCTTATCTCTAAGATGTGTCATTA TAGGTATTCAAAAATTCTAAAACTGGCAAAATTCTGAAATCTGAAGCACTTTTGCATAAAGGACTCAATC TCTGTGCTACTGCTGAATTGCTGCTCCGGGCTGGGGGCCTCATGTTCACTGACTTTGAAGTTATTTAATT ATCTCTTTTTCAATTCAGTTATTTTTTTTCACCAAAAAGGTAACAAATAGTTATTGATGAAATGAATCTT GTCTCTTTTCTCCAGATGCTTCAATAAATACCCCTAAATATTAATGAAGATTCCTGAACCATTCTAGGTG GGAGAAAGAAGAAGGAGCAGAGTCCCTATGTCCTTGTTCTAGGGTGACTGGATTGTAATTTGTGAATGCA GGTATAAAGTTAGAGAAATG

ENSMUST00000082908

gtgcttgcttcggcaacacatacactaaattttgaacgatacagagaagattagcatgggccctgcacaa ggatgacacacaaaATATAAATATTGAGTAACAAAAATAT

..................

roryk commented 5 years ago

Can you do a ls -l on the umis directory? Do the rapmap alignments look reasonable? What I think is happening is you are not quantitating any genes for these samples. The loss could be either, like Lorena suggested, the rapmap index is borked. But it could also be there are very few or no input reads or input alignments.

zji90 commented 5 years ago

Sure. Please see below for ls -lh results for umis directory. I guess the problem is probably due to the broken rapmap index because I was able to successfully finish the pipeline and get reasonable results using the default settings (without specifying transcriptome_gtf). Thus the data itself should be OK.

drwxr-xr-x 3 zji4@jhu.edu hji7 4.0K Feb 24 00:50 E14-2i-cond-2 drwxr-xr-x 2 zji4@jhu.edu hji7 4.0K Feb 24 04:03 E14-2i-cond-2-AGCTAGAA -rw-r--r-- 1 zji4@jhu.edu hji7 5.7G Feb 24 03:03 E14-2i-cond-2-AGCTAGAA.filtered.fq.gz -rw-r--r-- 1 zji4@jhu.edu hji7 36G Feb 23 15:00 E14-2i-cond-2.umitransformed.fq.gz drwxr-xr-x 3 zji4@jhu.edu hji7 4.0K Feb 23 19:55 E14-2i-D1 drwxr-xr-x 2 zji4@jhu.edu hji7 4.0K Feb 24 03:35 E14-2i-D1-ATTAGACG -rw-r--r-- 1 zji4@jhu.edu hji7 4.0G Feb 24 01:49 E14-2i-D1-ATTAGACG.filtered.fq.gz -rw-r--r-- 1 zji4@jhu.edu hji7 36G Feb 23 01:29 E14-2i-D1.umitransformed.fq.gz drwxr-xr-x 3 zji4@jhu.edu hji7 4.0K Feb 23 20:08 E14-2i-D3 drwxr-xr-x 2 zji4@jhu.edu hji7 4.0K Feb 24 03:41 E14-2i-D3-CTAGTCGA -rw-r--r-- 1 zji4@jhu.edu hji7 5.2G Feb 24 02:06 E14-2i-D3-CTAGTCGA.filtered.fq.gz -rw-r--r-- 1 zji4@jhu.edu hji7 36G Feb 23 01:27 E14-2i-D3.umitransformed.fq.gz drwxr-xr-x 3 zji4@jhu.edu hji7 4.0K Feb 24 00:56 E14-2i-D5 drwxr-xr-x 2 zji4@jhu.edu hji7 4.0K Feb 24 04:08 E14-2i-D5-ACTCTAGG -rw-r--r-- 1 zji4@jhu.edu hji7 5.6G Feb 24 03:16 E14-2i-D5-ACTCTAGG.filtered.fq.gz -rw-r--r-- 1 zji4@jhu.edu hji7 36G Feb 23 15:09 E14-2i-D5.umitransformed.fq.gz drwxr-xr-x 3 zji4@jhu.edu hji7 4.0K Feb 24 00:55 E14-2i-D7 drwxr-xr-x 2 zji4@jhu.edu hji7 4.0K Feb 24 03:59 E14-2i-D7-TCTTACGC -rw-r--r-- 1 zji4@jhu.edu hji7 3.7G Feb 24 02:55 E14-2i-D7-TCTTACGC.filtered.fq.gz -rw-r--r-- 1 zji4@jhu.edu hji7 36G Feb 23 14:59 E14-2i-D7.umitransformed.fq.gz drwxr-xr-x 3 zji4@jhu.edu hji7 4.0K Feb 23 20:00 E14-SL-cond-2 drwxr-xr-x 2 zji4@jhu.edu hji7 4.0K Feb 24 03:40 E14-SL-cond-2-CGGAGAGA -rw-r--r-- 1 zji4@jhu.edu hji7 4.9G Feb 24 02:00 E14-SL-cond-2-CGGAGAGA.filtered.fq.gz -rw-r--r-- 1 zji4@jhu.edu hji7 36G Feb 23 01:28 E14-SL-cond-2.umitransformed.fq.gz

roryk commented 5 years ago

Thanks-- that makes sense. Where did you get the transcriptome_gtf from?

zji90 commented 5 years ago

It's from gencode:

ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_mouse/release_M20/gencode.vM20.annotation.gtf.gz

roryk commented 5 years ago

If you look in the run that is broken for a tx2gene.csv file in the work directory, does that file have the transcript versions? (the .X at the end of the transcript name)

zji90 commented 5 years ago

Sorry I am not able to find a file named 'tx2gene.csv'. Is it under rapmap/index/ or under umis/?

roryk commented 5 years ago

No worries-- it should be under annotation in the work directory.

zji90 commented 5 years ago

Oh I don't see there is a directory called 'work' or 'annotation'. The existing directories are: 'bcbiotx', 'checkpoints_parallel', 'fastq', 'inputs', 'log', 'provenance', 'umis'

roryk commented 5 years ago

Hm. Is this the right directory? From the logs above:

/scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap

it looks like the rapmap directory should be there since it was failing after doing the umis fasttagcount.

zji90 commented 5 years ago

Oh yes the 'rapmap' folder is also there. I was looking at the wrong folder.

roryk commented 5 years ago

Great-- no annotation folder though?

zji90 commented 5 years ago

No. There is no 'annotation' folder.

roryk commented 5 years ago

Okey doke, I'll try to reproduce this tomorrow and fix it so you can use the Gencode GTF. Thanks!

zji90 commented 5 years ago

Thanks! Looking forward to the fix.

zji90 commented 5 years ago

I accidentally find the gencode.vM20.annotation-tx2gene.tsv file. It is in the same folder as the gtf file. The head of the file is as below.

ENSMUST00000119854.7 ENSMUSG00000037736.18 ENSMUST00000204196.1 ENSMUSG00000029804.17 ENSMUST00000053245.6 ENSMUSG00000013483.14 ENSMUST00000147408.2 ENSMUSG00000052516.19 ENSMUST00000033908.13 ENSMUSG00000031511.15 ENSMUST00000174016.7 ENSMUSG00000036036.15 ENSMUST00000082868.1 ENSMUSG00000064802.1 ENSMUST00000187399.1 ENSMUSG00000090925.2 ENSMUST00000194793.1 ENSMUSG00000103497.1 ENSMUST00000221298.1 ENSMUSG00000113688.1

roryk commented 5 years ago

Great-- this is what I thought the problem was. Could you also post the bcbio-nextgen-commands.log file?

zji90 commented 5 years ago

Sure. See below.

[2019-02-22T16:53Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fastqtransform --separate_cb /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-transform.json --cores 16 /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz| seqtk seq -L 20 - | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpBrvulx/E14-2i-D3.umitransformed.fq.gz [2019-02-22T16:53Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fastqtransform --separate_cb /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-transform.json --cores 16 /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz| seqtk seq -L 20 - | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpLbvI5m/E14-2i-D1.umitransformed.fq.gz [2019-02-22T16:53Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fastqtransform --separate_cb /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-transform.json --cores 16 /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz| seqtk seq -L 20 - | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp3ZtF0x/E14-SL-cond-2.umitransformed.fq.gz [2019-02-23T06:28Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fastqtransform --separate_cb /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-transform.json --cores 16 /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz| seqtk seq -L 20 - | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpmTjcfF/E14-2i-cond-2.umitransformed.fq.gz [2019-02-23T06:28Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fastqtransform --separate_cb /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-transform.json --cores 16 /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz| seqtk seq -L 20 - | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpRkDlex/E14-2i-D5.umitransformed.fq.gz [2019-02-23T06:30Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fastqtransform --separate_cb /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-transform.json --cores 16 /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R1.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R2.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R3.fq.gz /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/E14-2i_R4.fq.gz| seqtk seq -L 20 - | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpav4Wuv/E14-2i-D7.umitransformed.fq.gz [2019-02-23T20:09Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis demultiplex_samples --nedit 1 --barcodes /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-3.txt --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpJTlFwz/demultiplexed /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D3.umitransformed.fq.gz [2019-02-23T20:09Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis demultiplex_samples --nedit 1 --barcodes /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-1.txt --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp6Vobmi/demultiplexed /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1.umitransformed.fq.gz [2019-02-23T20:09Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis demultiplex_samples --nedit 1 --barcodes /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-2.txt --outdir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpJcdGU/demultiplexed /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-SL-cond-2.umitransformed.fq.gz [2019-02-24T00:55Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis demultiplex_samples --nedit 1 --barcodes /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-4.txt --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp5CoaK2/demultiplexed /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-cond-2.umitransformed.fq.gz [2019-02-24T01:00Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis demultiplex_samples --nedit 1 --barcodes /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-5.txt --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp1KbnKm/demultiplexed /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D5.umitransformed.fq.gz [2019-02-24T01:08Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis demultiplex_samples --nedit 1 --barcodes /home-4/zji4@jhu.edu/scratch/mousereprogram/data/raw/sample-index-6.txt --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpDLWSOa/demultiplexed /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D7.umitransformed.fq.gz [2019-02-24T05:56Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_filter --cores 16 --bc1 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb1.txt.gz --nedit 1 --bc2 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb2.txt.gz /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1/demultiplexed/ATTAGACG.fq | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmprehLfH/E14-2i-D1-ATTAGACG.filtered.fq.gz [2019-02-24T05:56Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_filter --cores 16 --bc1 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb1.txt.gz --nedit 1 --bc2 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb2.txt.gz /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-SL-cond-2/demultiplexed/CGGAGAGA.fq | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp1ujDFO/E14-SL-cond-2-CGGAGAGA.filtered.fq.gz [2019-02-24T05:56Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cbfilter --cores 16 --bc1 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb1.txt.gz --nedit 1 --bc2 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb2.txt.gz /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D3/demultiplexed/CTAGTCGA.fq | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpCbuFf/E14-2i-D3-CTAGTCGA.filtered.fq.gz [2019-02-24T06:49Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_filter --cores 16 --bc1 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb1.txt.gz --nedit 1 --bc2 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb2.txt.gz /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-cond-2/demultiplexed/AGCTAGAA.fq | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpnspIUh/E14-2i-cond-2-AGCTAGAA.filtered.fq.gz [2019-02-24T07:00Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_filter --cores 16 --bc1 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb1.txt.gz --nedit 1 --bc2 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb2.txt.gz /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D5/demultiplexed/ACTCTAGG.fq | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp6n_GOg/E14-2i-D5-ACTCTAGG.filtered.fq.gz [2019-02-24T07:06Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_filter --cores 16 --bc1 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb1.txt.gz --nedit 1 --bc2 /home-4/zji4@jhu.edu/scratch/software/bcbio/anaconda/lib/python2.7/site-packages/bcbio/rnaseq/../data/umis/harvard-indrop-v3-cb2.txt.gz /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D7/demultiplexed/TCTTACGC.fq | gzip > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpCYjVuW/E14-2i-D7-TCTTACGC.filtered.fq.gz [2019-02-24T08:16Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1-ATTAGACG.filtered.fq.gz > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp3ALlib/cb-histogram.txt [2019-02-24T08:16Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-SL-cond-2-CGGAGAGA.filtered.fq.gz > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmphwodPN/cb-histogram.txt [2019-02-24T08:16Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D3-CTAGTCGA.filtered.fq.gz > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpM59wiM/cb-histogram.txt [2019-02-24T08:35Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-cond-2-AGCTAGAA.filtered.fq.gz > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpm31o3m/cb-histogram.txt [2019-02-24T08:40Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D5-ACTCTAGG.filtered.fq.gz > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpooWH70/cb-histogram.txt [2019-02-24T08:41Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D7-TCTTACGC.filtered.fq.gz > /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpcfdpPv/cb-histogram.txt [2019-02-24T09:08Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis kallisto --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D3-CTAGTCGA/cb-histogram.txt --cb_cutoff 1000 --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpgGhr5S/fastq /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D3-CTAGTCGA.filtered.fq.gz [2019-02-24T09:08Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis kallisto --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-SL-cond-2-CGGAGAGA/cb-histogram.txt --cb_cutoff 1000 --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpVevvnX/fastq /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-SL-cond-2-CGGAGAGA.filtered.fq.gz [2019-02-24T09:08Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis kallisto --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1-ATTAGACG/cb-histogram.txt --cb_cutoff 1000 --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpG_CwxK/fastq /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1-ATTAGACG.filtered.fq.gz [2019-02-24T09:35Z] gffread -g /scratch/users/zji4@jhu.edu/software/bcbio/genomes/Mmusculus/mm10/seq/mm10.fa -w /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpmTGWPJ/mm10.fa.tmp /scratch/users/zji4@jhu.edu/software/bcbio/genomes/Mmusculus/mm10/rnaseq/ref-transcripts.gtf [2019-02-24T09:36Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/kallisto index -k 31 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpXOFvh1/mm10.idx /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/inputs/transcriptome/mm10.fa [2019-02-24T09:40Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/kallisto index -k 31 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpxPfTXv/mm10.idx /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/inputs/transcriptome/mm10.fa [2019-02-24T09:41Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/kallisto index -k 31 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp_jvOcV/mm10.idx /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/inputs/transcriptome/mm10.fa [2019-02-24T09:42Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/kallisto pseudo --umi -t 16 -o /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/kallisto/E14-2i-D1-ATTAGACG/fastq/bcbiotx/tmpK_H0eb/quant -b /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/kallisto/E14-2i-D1-ATTAGACG/fastq/barcodes.batch -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/kallisto/E14-2i-D1-ATTAGACG/index/mm10/mm10.idx [2019-02-24T09:43Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis kallisto --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-cond-2-AGCTAGAA/cb-histogram.txt --cb_cutoff 1000 --out_dir /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPs5Yqa/fastq /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-cond-2-AGCTAGAA.filtered.fq.gz [2019-02-24T11:33Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/rapmap quasiindex -k 31 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpRQjAn5/mm10 -t /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/index/quasiindex/mm10/mm10.fa [2019-02-24T11:36Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/rapmap quasimap -t 16 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/index/quasiindex/mm10 -r <(gzip -cd /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1-ATTAGACG.filtered.fq.gz) | /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools sort -@ 16 -m 1G -T /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpA_XFEO/E14-2i-D1-ATTAGACG-sorttmp -o /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpA_XFEO/E14-2i-D1-ATTAGACG.bam /dev/stdin [2019-02-24T11:36Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/rapmap quasimap -t 16 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/index/quasiindex/mm10 -r <(gzip -cd /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-SL-cond-2-CGGAGAGA.filtered.fq.gz) | /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools sort -@ 16 -m 1G -T /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp684S5c/E14-SL-cond-2-CGGAGAGA-sorttmp -o /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp684S5c/E14-SL-cond-2-CGGAGAGA.bam /dev/stdin [2019-02-24T11:36Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/rapmap quasimap -t 16 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/index/quasiindex/mm10 -r <(gzip -cd /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D3-CTAGTCGA.filtered.fq.gz) | /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools sort -@ 16 -m 1G -T /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpVXlMri/E14-2i-D3-CTAGTCGA-sorttmp -o /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpVXlMri/E14-2i-D3-CTAGTCGA.bam /dev/stdin [2019-02-24T11:42Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools index -@ 16 /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D1-ATTAGACG/E14-2i-D1-ATTAGACG.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpu6ioAB/E14-2i-D1-ATTAGACG.bam.bai [2019-02-24T11:43Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/rapmap quasimap -t 16 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/index/quasiindex/mm10 -r <(gzip -cd /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-cond-2-AGCTAGAA.filtered.fq.gz) | /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools sort -@ 16 -m 1G -T /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpyOw_B3/E14-2i-cond-2-AGCTAGAA-sorttmp -o /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpyOw_B3/E14-2i-cond-2-AGCTAGAA.bam /dev/stdin [2019-02-24T11:44Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools index -@ 16 /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-SL-cond-2-CGGAGAGA/E14-SL-cond-2-CGGAGAGA.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpBYwiJc/E14-SL-cond-2-CGGAGAGA.bam.bai [2019-02-24T11:45Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools index -@ 16 /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D3-CTAGTCGA/E14-2i-D3-CTAGTCGA.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp_tOxM3/E14-2i-D3-CTAGTCGA.bam.bai [2019-02-24T11:45Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/rapmap quasimap -t 16 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/index/quasiindex/mm10 -r <(gzip -cd /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D5-ACTCTAGG.filtered.fq.gz) | /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools sort -@ 16 -m 1G -T /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmps5W6Zd/E14-2i-D5-ACTCTAGG-sorttmp -o /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmps5W6Zd/E14-2i-D5-ACTCTAGG.bam /dev/stdin [2019-02-24T11:46Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/rapmap quasimap -t 16 -i /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/index/quasiindex/mm10 -r <(gzip -cd /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D7-TCTTACGC.filtered.fq.gz) | /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools sort -@ 16 -m 1G -T /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpxsklWY/E14-2i-D7-TCTTACGC-sorttmp -o /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpxsklWY/E14-2i-D7-TCTTACGC.bam /dev/stdin [2019-02-24T11:52Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools index -@ 16 /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D7-TCTTACGC/E14-2i-D7-TCTTACGC.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpSTMqv8/E14-2i-D7-TCTTACGC.bam.bai [2019-02-24T11:52Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools index -@ 16 /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-cond-2-AGCTAGAA/E14-2i-cond-2-AGCTAGAA.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpaoTGqE/E14-2i-cond-2-AGCTAGAA.bam.bai [2019-02-24T11:53Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/samtools index -@ 16 /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D5-ACTCTAGG/E14-2i-D5-ACTCTAGG.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp9mPmxo/E14-2i-D5-ACTCTAGG.bam.bai [2019-02-24T11:55Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fasttagcount --cb_cutoff 1000 --genemap /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation-tx2gene.tsv --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-SL-cond-2-CGGAGAGA/cb-histogram.txt --umi_matrix /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpJMRg7w/E14-SL-cond-2-CGGAGAGA-dupes.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-SL-cond-2-CGGAGAGA/E14-SL-cond-2-CGGAGAGA.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpJMRg7w/E14-SL-cond-2-CGGAGAGA.mtx.full [2019-02-24T11:55Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fasttagcount --cb_cutoff 1000 --genemap /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation-tx2gene.tsv --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D3-CTAGTCGA/cb-histogram.txt --umi_matrix /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpuUQ9qL/E14-2i-D3-CTAGTCGA-dupes.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D3-CTAGTCGA/E14-2i-D3-CTAGTCGA.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpuUQ9qL/E14-2i-D3-CTAGTCGA.mtx.full [2019-02-24T11:55Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fasttagcount --cb_cutoff 1000 --genemap /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation-tx2gene.tsv --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1-ATTAGACG/cb-histogram.txt --umi_matrix /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp0lRNvo/E14-2i-D1-ATTAGACG-dupes.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D1-ATTAGACG/E14-2i-D1-ATTAGACG.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp0lRNvo/E14-2i-D1-ATTAGACG.mtx.full [2019-02-24T11:58Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp0lRNvo/E14-2i-D1-ATTAGACG.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmp0lRNvo/E14-2i-D1-ATTAGACG.mtx [2019-02-24T11:58Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpuUQ9qL/E14-2i-D3-CTAGTCGA.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpuUQ9qL/E14-2i-D3-CTAGTCGA.mtx [2019-02-24T11:58Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpJMRg7w/E14-SL-cond-2-CGGAGAGA.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpJMRg7w/E14-SL-cond-2-CGGAGAGA.mtx [2019-02-24T12:29Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis fasttagcount --cb_cutoff 1000 --genemap /home-4/zji4@jhu.edu/scratch/mousereprogram/data/gtf/gencode.vM20.annotation-tx2gene.tsv --cb_histogram /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/umis/E14-2i-D1-ATTAGACG/cb-histogram.txt --umi_matrix /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG-dupes.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/rapmap/E14-2i-D1-ATTAGACG/E14-2i-D1-ATTAGACG.bam /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full [2019-02-24T12:32Z] /scratch/users/zji4@jhu.edu/software/bcbio/galaxy/../anaconda/bin/umis sparse /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx.full /scratch/users/zji4@jhu.edu/mousereprogram/data/align/run2/bcbiotx/tmpPW0qyj/E14-2i-D1-ATTAGACG.mtx

roryk commented 5 years ago

Thanks so much and sorry for the problem. I could reproduce it using that gencode file. The problem is that there are no txgene mappings which are valid. Our code to detect if transcripts versions exist (and strip them off) in the GTF file assumed there was an entry transcript_version in the GTF attributes field which held the number, but in the Gencode GTF it is already part of the transcript_id. I fixed that so it should produce the right mappings now. I also fixed the issue where we were writing the tx2gene file in the GTF directory instead of the analysis directory. I think you should be all set now to use the GENCODE file.

roryk commented 5 years ago

If you run:

bcbio_nextgen.py upgrade -u development

you'll get the updated version with the fix. Please re-open the issue if it didn't actually fix your problem. Sorry again for the issue!