search

velocyto-team / velocyto.py

RNA velocity estimation in Python
http://velocyto.org/velocyto.py/
BSD 2-Clause "Simplified" License
159 stars 84 forks source link

Issues running velocyto #163

Open shabs24 opened 5 years ago

shabs24 commented 5 years ago

Hi,

I am trying to run velocyto using both run and run10x command. I get the same error in each case. It looks like an issue with samtools

I am running it on cluster using 1 node and 12ppn. I am not sure how to set the parameters in veloctyto for samtools on a cluster

I tried to run samtools on passorted_bam.bam to generate cell sorted file and try to run velocyto on it and I still get same error.

See the commands and error below

Any help will be appreciated!

Many Thanks

run10x

velocyto run10x -m /cluster/projects//mm10_rmsk.gtf /cluster/projects/MFS_DCS_pool /cluster/projects//genes.gtf

run

velocyto run -b outs/filtered_gene_bc_matrices/mm10/barcodes.tsv -o ./velocyto_run -m /cluster/projects//mm10_rmsk.gtf outs/possorted_genome_bam.bam /cluster/projects//genes.gtf

run -> after sort

samtools sort -l 0 -O BAM -t CB outs/possorted_genome_bam.bam -o cellsorted_possorted_genome_bam.bam velocyto run -b outs/filtered_gene_bc_matrices/mm10/barcodes.tsv -o ./velocyto_run_cellsorted -m /cluster/projects//mm10_rmsk.gtf cellsorted_possorted_genome_bam.bam /cluster/projects//genes.gtf ############################## And I get the following error ############################## 018-12-14 15:32:12,809 - DEBUG - Read first 290 million reads 2018-12-14 15:32:31,928 - DEBUG - End of file. Reset index: start scanning from initial position. 2018-12-14 15:32:31,965 - DEBUG - 2742974 reads were skipped because no apropiate cell or umi barcode was found 2018-12-14 15:32:31,966 - INFO - Now just waiting that the bam sorting process terminates

samtools sort: couldn't allocate memory for bam_mem /cluster/tools/software/python/3.6.5/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Traceback (most recent call last): File "/cluster/tools/software/velocyto/0.17.13/bin/velocyto", line 11, in sys.exit(cli()) File "/cluster/tools/software/velocyto/0.17.13/lib/python3.6/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/cluster/tools/software/velocyto/0.17.13/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/cluster/tools/software/velocyto/0.17.13/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/cluster/tools/software/velocyto/0.17.13/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/cluster/tools/software/velocyto/0.17.13/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/cluster/tools/software/velocyto/0.17.13/lib/python3.6/site-packages/velocyto/commands/run10x.py", line 106, in run10x samtools_memory=samtools_memory, dump=dump, verbose=verbose, additional_ca=additional_ca) File "/cluster/tools/software/velocyto/0.17.13/lib/python3.6/site-packages/velocyto/commands/_run.py", line 225, in _run Otherwise sort manually by samtools sort -l [compression] -m [mb_to_use]M -t [tagname] -O BAM -@ [threads_to_use] -o cellsorted_[bamfile] [bamfile]") MemoryError: bam file #0 could not be sorted by cells. This is probably related to an old version of samtools, please install samtools >= 1.6. In alternative this could be a memory error, try to set the --samtools_memory option to a value compatible with your system. Otherwise sort manually by samtools sort -l [compression] -m [mb_to_use]M -t [tagname] -O BAM -@ [threads_to_use] -o cellsorted_[bamfile] [bamfile]

idalarsson commented 5 years ago

@shabs24 Did you manage to solve this? I receive the same error.

YingMa0107 commented 5 years ago

@shabs24 @idalarsson Did you solve this? I also got the same error when I tried run10X, run after sort both.

shabs24 commented 5 years ago

@shabs24 @idalarsson Did you solve this? I also got the same error when I tried run10X, run after sort both.

@idalarsson @YingMa1993 I was running on cluster and was able to solve by requesting more memory (60 GB).

davisidarta commented 5 years ago

@shabs24 , how much memory do you have avaiable on your cluster? I also got the same error. The file is 10x, So I suppose it's already sorted. I'm running it on a 128GB RAM system.

shabs24 commented 5 years ago

@davisidarta, I have used 60 Gb memory on the cluster.

ChelseaCHENX commented 5 years ago

I had the same problem ... how many threads are you using? (want an estimation of threads ~ memory per sample)

saeedfc commented 5 years ago

I am using 240 GB for a 1000 cells and still hitting the same error. No idea what is going wrong. Samtools is version 1.9 ~$ velocyto run10x -m /alltracks_mask.gtf /KUL-1-1000cells/ /refdata-cellranger-mm10-3.0.0/genes/genes.gtf --samtools-memory 240000

MemoryError: bam file #0 could not be sorted by cells.
                This is probably related to an old version of samtools, please install samtools >= 1.6.                In alternative this could be a memory error, try to set the --samtools_memory option to a value compatible with your system.                 Otherwise sort manually by samtools ``sort -l [compression] -m [mb_to_use]M -t [tagname] -O BAM -@ [threads_to_use] -o cellsorted_[bamfile] [bamfile]``
saeedfc commented 5 years ago

If anyone is struggling with the same issue, please see this

https://github.com/velocyto-team/velocyto.py/issues/192#issuecomment-545349435

denvercal1234GitHub commented 2 years ago

@saeedfc @shabs24 After you cellsort the possort bam file, did you just use the same bam.bai index file that came with the possort bam from cellranger? Or, did you index the cellsorted_possorted bam? Thank you so much! --- issue #321

linxy29 commented 2 years ago

I faced the same error, I tried requesting more memory and it didn't work. Since I used conda, so I created a new environment and installed velocyto based on the installation guide. The problem is solved.

hanjun98 commented 1 year ago

nowadays, I can shoot this problem like this

1.create new env

conda create -n velocyto python=3.9

2.install samtools, first(maybe you can get a samtools version =1.17

conda install -c bioconda samtools

3.and then, install other things(reference by https://velocyto.org/velocyto.py/install/index.html)

conda install numpy scipy cython numba matplotlib scikit-learn h5py click

4.install velocyto through this way(reference by https://velocyto.org/velocyto.py/install/index.html). Because when I do"pip install velocyto", It has a error like version collision with other package(ex:pandas,,etc.)

git clone https://github.com/velocyto-team/velocyto.py.git

cd velocyto.py

pip install -e .

  1. and then excute your velocyto code

velocyto run10x -m /cluster/projects//mm10_rmsk.gtf /cluster/projects/MFS_DCS_pool /cluster/projects//genes.gtf

***or you can try this way that recommend by this website(https://lightrun.com/answers/velocyto-team-velocyto-py-issues-running-velocyto). This website told you that run samtools, first. And then, run velocyto.

good luck:))

maminby commented 6 months ago

SOLVED！ Run velocyto begin with velocyto run --samtools-memory 4000 this will set your samtools memory to 4000MB, which is based on the instruction of this:https://velocyto.org/velocyto.py/fullapi/cliapi.html?highlight=memory#cmdoption-velocyto-run-samtools-memory.