velocyto-team / velocyto.py

RNA velocity estimation in Python
http://velocyto.org/velocyto.py/
BSD 2-Clause "Simplified" License
159 stars 83 forks source link

10X sample analysis #167

Closed somnathtagore closed 5 years ago

somnathtagore commented 5 years ago

Hi,

I am testing the tool using a dataset provided by 10X genomics (https://support.10xgenomics.com/single-cell-gene-expression/datasets/2.1.0/neurons_900)

I am facing the following error. Can you please help?

~/anaconda3/envs/my_pymc_env/bin/velocyto run -o /.../rna-velocity/samples -m /.../rna-velocity/samples/mm10_rmsk.gtf -v /.../rna-velocity/samples/neurons_900_possorted_genome_bam.bam /.../rna-velocity/samples/genes_gtf/Mus_musculus.GRCm38.95.gtf 2019-01-10 12:57:08,469 - INFO - No SAMPLEID specified, the sample will be called neurons_900_possorted_genome_bam_Y5VZK (last 5 digits are a random-id to avoid overwriting some other file by mistake) 2019-01-10 12:57:08,469 - DEBUG - Using logic: Default 2019-01-10 12:57:08,469 - DEBUG - Cell barcodes will be determined while reading the .bam file 2019-01-10 12:57:08,489 - WARNING - Your system does not support calling grep MemAvailable /proc/meminfo so the memory effort for the samtools command could not be chosen appropriately. 32Gb will be assumed 2019-01-10 12:57:08,489 - DEBUG - Peeking into /.../rna-velocity/samples/neurons_900_possorted_genome_bam.bam 2019-01-10 12:57:08,533 - WARNING - Not found cell and umi barcode in entry 27 of the bam file 2019-01-10 12:57:08,533 - WARNING - Not found cell and umi barcode in entry 36 of the bam file 2019-01-10 12:57:08,535 - WARNING - Not found cell and umi barcode in entry 134 of the bam file 2019-01-10 12:57:08,537 - WARNING - Not found cell and umi barcode in entry 264 of the bam file 2019-01-10 12:57:08,537 - WARNING - Not found cell and umi barcode in entry 281 of the bam file 2019-01-10 12:57:08,538 - WARNING - Not found cell and umi barcode in entry 282 of the bam file 2019-01-10 12:57:08,540 - WARNING - Not found cell and umi barcode in entry 353 of the bam file 2019-01-10 12:57:08,540 - WARNING - Not found cell and umi barcode in entry 354 of the bam file 2019-01-10 12:57:08,541 - WARNING - Not found cell and umi barcode in entry 355 of the bam file 2019-01-10 12:57:08,542 - WARNING - Not found cell and umi barcode in entry 431 of the bam file 2019-01-10 12:57:08,544 - WARNING - Not found cell and umi barcode in entry 522 of the bam file 2019-01-10 12:57:08,544 - WARNING - Not found cell and umi barcode in entry 523 of the bam file 2019-01-10 12:57:08,544 - WARNING - Not found cell and umi barcode in entry 524 of the bam file 2019-01-10 12:57:08,545 - WARNING - Not found cell and umi barcode in entry 638 of the bam file 2019-01-10 12:57:08,545 - WARNING - Not found cell and umi barcode in entry 668 of the bam file 2019-01-10 12:57:08,545 - WARNING - Not found cell and umi barcode in entry 693 of the bam file 2019-01-10 12:57:08,546 - WARNING - Not found cell and umi barcode in entry 761 of the bam file 2019-01-10 12:57:08,547 - WARNING - Not found cell and umi barcode in entry 926 of the bam file 2019-01-10 12:57:08,548 - WARNING - Not found cell and umi barcode in entry 979 of the bam file 2019-01-10 12:57:08,548 - WARNING - Not found cell and umi barcode in entry 980 of the bam file 2019-01-10 12:57:08,548 - WARNING - Not found cell and umi barcode in entry 982 of the bam file 2019-01-10 12:57:08,548 - WARNING - Not found cell and umi barcode in entry 992 of the bam file 2019-01-10 12:57:08,548 - WARNING - Not found cell and umi barcode in entry 994 of the bam file 2019-01-10 12:57:08,548 - WARNING - Not found cell and umi barcode in entry 995 of the bam file 2019-01-10 12:57:08,549 - WARNING - Not found cell and umi barcode in entry 996 of the bam file 2019-01-10 12:57:08,549 - WARNING - Not found cell and umi barcode in entry 1005 of the bam file 2019-01-10 12:57:08,552 - WARNING - The file /.../rna-velocity/samples/cellsorted_neurons_900_possorted_genome_bam.bam already exists. The sorting step will be skipped and the existing file will be used. 2019-01-10 12:57:08,552 - INFO - Load the annotation from /.../rna-velocity/genes_gtf/Mus_musculus.GRCm38.95.gtf 2019-01-10 12:57:08,957 - WARNING - The entry exon_number was not present in the gtf file. It will be infferred from the position. Traceback (most recent call last): File "/ifs/scratch/c2b2/ac_lab/st3179/rna-velocity/velocyto.py-master/velocyto/counter.py", line 586, in peek_and_correct trid = regex_trid.search(tags).group(1) AttributeError: 'NoneType' object has no attribute 'group'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/.../anaconda3/envs/my_pymc_env/bin/velocyto", line 11, in load_entry_point('velocyto', 'console_scripts', 'velocyto')() File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/.../rna-velocity/velocyto.py-master/velocyto/commands/run.py", line 113, in run samtools_memory=samtools_memory, dump=dump, verbose=verbose, additional_ca=additional_ca) File "/.../rna-velocity/velocyto.py-master/velocyto/commands/_run.py", line 186, in _run annotations_by_chrm_strand = exincounter.read_transcriptmodels(gtffile) File "/.../rna-velocity/velocyto.py-master/velocyto/counter.py", line 468, in read_transcriptmodels gtf_lines = self.peek_and_correct(gtf_lines) File "/.../rna-velocity/velocyto.py-master/velocyto/counter.py", line 588, in peek_and_correct raise AttributeError(f"transcript_id entry not found in line: {lin}") AttributeError: transcript_id entry not found in line: 1 ensembl exon 12190 12227 . + .

Many thanks!

gioelelm commented 5 years ago

The gtf file is not formatted in an adequate way (it is missing transcript id) and is therefore incompatible with velocyto... have you checked carefully this section of the tutorial: http://velocyto.org/velocyto.py/tutorial/cli.html#requirements-on-the-input-files Let me know if after fixing the gtf the problem persists.

somnathtagore commented 5 years ago

Hi, thank you for your suggestions. the script is now working... but, i get something like this at the end:

2019-01-17 00:14:50,626 - DEBUG - Writing loom file /.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/matplotlib/font_manager.py:232: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. 'Matplotlib is building the font cache using fc-list. ' Traceback (most recent call last): File "/.../rna-velocity/velocyto.py-master/velocyto/commands/_run.py", line 286, in _run ds = loompy.create(filename=outfile, matrix=total, row_attrs=ra, col_attrs=ca, dtype="float32") TypeError: create() got an unexpected keyword argument 'matrix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/.../anaconda3/envs/my_pymc_env/bin/velocyto", line 11, in load_entry_point('velocyto', 'console_scripts', 'velocyto')() File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/.../rna-velocity/velocyto.py-master/velocyto/commands/run.py", line 113, in run samtools_memory=samtools_memory, dump=dump, verbose=verbose, additional_ca=additional_ca) File "/.../rna-velocity/velocyto.py-master/velocyto/commands/_run.py", line 297, in _run loompy.create(filename=outfile, layers=tmp_layers, row_attrs=ra, col_attrs=ca, file_attrs={"velocyto.version": vcy.version, "velocyto.logic": logic}) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/loompy/loompy.py", line 989, in create with new(filename, file_attrs=file_attrs) as ds: File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/loompy/loompy.py", line 918, in new f = h5py.File(name=filename, mode='w') File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in init swmr=swmr) File "/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/h5py/_hl/files.py", line 176, in make_fid fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 105, in h5py.h5f.create OSError: Unable to create file (unable to lock file, errno = 37, error message = 'No locks available')

gioelelm commented 5 years ago

This is a problem related to your loompy or h5py installation not velocyto.

somnathtagore commented 5 years ago

Hi, I installed h5py using conda (conda install -c anaconda h5py) and loompy (conda install -c conda-forge loompy). and checked whether they are installed or not:

$ python Python 3.7.1 (default, Dec 14 2018, 19:28:38) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

import h5py import loompy quit()

Then, i removed the output folder with the empty loom file... but still, i get an empty loom file: -rw-r--r--. 1 .... ... 0 Jan 18 21:55 neurons_900_possorted_genome_bam_WEF5N.loom

Can you suggest anything? Many thanks!

gioelelm commented 5 years ago

Does the command run all the way without error this time?

Anyways your test is not sufficient to check that loompy or h5py work. You should create a new file to be sure.

I cannot help you so much with the little info I have. Could you also check it with some other dataset, maybe it is dataset specific?

slinnarsson commented 5 years ago

Hi

Installing loompy from conda-forge will give you an old version, which may or may not work. Please try with the officially supported installation method, pip install -U loompy (see http://linnarssonlab.org/loompy/installation/index.html).

@gioelelm do you know how to work with conda-forge? How does it find packages? The loompy there is old and probably very broken, so we should figure out how to either remove it or make sure it's always up to date, but I can't figure out how it got there in the first place.

somnathtagore commented 5 years ago

Hi,

I tried installing loompy using pip install -U loompy, but it gives an error. That is why I used conda to install loompy.

$ pip install -U loompy Collecting loompy Using cached https://files.pythonhosted.org/packages/18/38/6231e36536a3741e2cc10fada493b8f1d900a57f13f6b416528a195fc601/loompy-2.0.12.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-rjmdnn54/loompy/setup.py", line 26 download_url=f"https://github.com/linnarsson-lab/loompy/archive/{__version__}.tar.gz", ^ SyntaxError: invalid syntax

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-rjmdnn54/loompy/

Many thanks! Som

Somnath Tagore, Ph.D. Califano Laboratory of Systems Biology Columbia University Medical Center Herbert Irving Cancer Research Center 1130 St. Nicholas Avenue New York, NY 10032, USA

From: Sten Linnarsson notifications@github.com Reply-To: "velocyto-team/velocyto.py" reply@reply.github.com Date: Tuesday, January 22, 2019 at 10:06 AM To: "velocyto-team/velocyto.py" velocyto.py@noreply.github.com Cc: "Tagore, Somnath" st3179@cumc.columbia.edu, Author author@noreply.github.com Subject: Re: [velocyto-team/velocyto.py] 10X sample analysis (#167)

Hi

Installing loompy from conda-forge will give you an old version, which may or may not work. Please try with the officially supported installation method, pip install -U loompy (see http://linnarssonlab.org/loompy/installation/index.htmlhttps://urldefense.proofpoint.com/v2/url?u=http-3A__linnarssonlab.org_loompy_installation_index.html&d=DwMFaQ&c=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U&r=Lk-l_Zb3ZiqAHwVArS-fulgioJpm3zXp-Qb7n8KVLOA&m=7KDiWZ8hatNWqM3zGaj10x78RJI3oUIbFjnTYUD5GJs&s=gUj96DcLW22MBa0y0q1CMHrdQDRDponMxG1jmYnV0dY&e=).

@gioelelmhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_gioelelm&d=DwMFaQ&c=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U&r=Lk-l_Zb3ZiqAHwVArS-fulgioJpm3zXp-Qb7n8KVLOA&m=7KDiWZ8hatNWqM3zGaj10x78RJI3oUIbFjnTYUD5GJs&s=UgvCZ0zIpzGlQge8Ev-pdMUHrIb7bB5mp5lBvQXFdK4&e= do you know how to work with conda-forge? How does it find packages? The loompy there is old and probably very broken, so we should figure out how to either remove it or make sure it's always up to date, but I can't figure out how it got there in the first place.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_velocyto-2Dteam_velocyto.py_issues_167-23issuecomment-2D456432365&d=DwMFaQ&c=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U&r=Lk-l_Zb3ZiqAHwVArS-fulgioJpm3zXp-Qb7n8KVLOA&m=7KDiWZ8hatNWqM3zGaj10x78RJI3oUIbFjnTYUD5GJs&s=5s6a47eWPF69ZS9aSD_gvYBMholeGChgSEKVuRl6_4c&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AhMdHYfPsyK0Z7SF2CcqtBfzsCIkoQ0Bks5vFyjQgaJpZM4Z6CFh&d=DwMFaQ&c=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U&r=Lk-l_Zb3ZiqAHwVArS-fulgioJpm3zXp-Qb7n8KVLOA&m=7KDiWZ8hatNWqM3zGaj10x78RJI3oUIbFjnTYUD5GJs&s=V4mAiSpWbCFYyedxPl4ufEEbF2cQL2ZmuZuLFpZGsj0&e=.

slinnarsson commented 5 years ago

Check the installation instructions again - loompy requires Python 3.6. You are using an older python.

gioelelm commented 5 years ago

@slinnarsson yes I put it there, there is this feedstock repo here: https://github.com/conda-forge/loompy-feedstock/blob/master/recipe/meta.yaml. To update the conda-forge version the recipe needs to be updated. When it gets updated, upon commit a rebuild of all the conda-forge repo gets triggered and, if the checks pass, the package gets added to conda-forge.

Updating the recipe (assuming the requirements stay the same) consists in changing the name of the version and updating the sha256 checksum of the source files .zip that gets uploaded to pip when releasing a new version (see here https://pypi.org/project/loompy/#files).

So yes I haven't kept updating the conda-forge. Ideally the way I meant it to work was that that upon the commit of a new "tagged" release version of loompy the TravisCI of the loompy repo would be triggered, the Travis server would not only release it on pip automatically (is this in place anymore? for velocyto it is) but also change the meta.yml recipe on the loompy-feedstock repo and therefore trigger in turn the update the conda version as well. Unfortunately I never figured out the script to do this second part and left it behind at some point.

gioelelm commented 5 years ago

Check the installation instructions again - loompy requires Python 3.6. You are using an older python.

Are you sure ? He reported this above:

$ python Python 3.7.1 (default, Dec 14 2018, 19:28:38) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

import h5py import loompy quit()

slinnarsson commented 5 years ago

But the syntax error is for an f-string, which indicates a lower version. Maybe it’s not the right pip?

Can you do ’which pip’ and ’which conda’ to make sure you have Anaconda’s pip? And let us know the results.

gioelelm commented 5 years ago

Yes very likely one of these very complicated problems that arise when one installs anaconda 2.7 and then upgrades it / makes a env with python3.... One needs to be very careful of which pip is one calling then.

somnathtagore commented 5 years ago

Hi,

Yes, you were correct. My pip and conda version was not same. Now, I have this:

$ which conda

~/anaconda3/bin/conda

$ which pip ~/anaconda3/envs/my_pymc_env/bin/pip

Now, I reinstalled loompy using: $ pip install -U loompy

I am now running velocyto again as pointed out by @gioelelmhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_gioelelm&d=DwMFaQ&c=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U&r=Lk-l_Zb3ZiqAHwVArS-fulgioJpm3zXp-Qb7n8KVLOA&m=7KDiWZ8hatNWqM3zGaj10x78RJI3oUIbFjnTYUD5GJs&s=UgvCZ0zIpzGlQge8Ev-pdMUHrIb7bB5mp5lBvQXFdK4&e=

I will keep you posted with the result.

Many thanks! Som

Somnath Tagore, Ph.D. Califano Laboratory of Systems Biology Columbia University Medical Center Herbert Irving Cancer Research Center 1130 St. Nicholas Avenue New York, NY 10032, USA

From: Sten Linnarsson notifications@github.com Reply-To: "velocyto-team/velocyto.py" reply@reply.github.com Date: Tuesday, January 22, 2019 at 10:38 AM To: "velocyto-team/velocyto.py" velocyto.py@noreply.github.com Cc: "Tagore, Somnath" st3179@cumc.columbia.edu, Author author@noreply.github.com Subject: Re: [velocyto-team/velocyto.py] 10X sample analysis (#167)

But the syntax error is for an f-string, which indicates a lower version. Maybe it’s not the right pip?

Can you do ’which pip’ and ’which conda’ to make sure you have Anaconda’s pip? And let us know the results.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_velocyto-2Dteam_velocyto.py_issues_167-23issuecomment-2D456444825&d=DwMFaQ&c=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U&r=Lk-l_Zb3ZiqAHwVArS-fulgioJpm3zXp-Qb7n8KVLOA&m=uYMO76W_Shff-AXGVuVW0-Hrx-PO9kZB0GP4jJnIIL4&s=q5mGXU4PyhTKNuxlJJej_vhrU9WdaAmmsiXcXpRGMrE&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AhMdHbL1-5FJkMzplqCU-2D3oMfY1v5zfQ1Iks5vFzBLgaJpZM4Z6CFh&d=DwMFaQ&c=G2MiLlal7SXE3PeSnG8W6_JBU6FcdVjSsBSbw6gcR0U&r=Lk-l_Zb3ZiqAHwVArS-fulgioJpm3zXp-Qb7n8KVLOA&m=uYMO76W_Shff-AXGVuVW0-Hrx-PO9kZB0GP4jJnIIL4&s=9vZUSoVFQAFmPvZ21Q_Chy2LExMEVx-QQg9PU2tbhfU&e=.

somnathtagore commented 5 years ago

Hi,

i did all the steps, $ which conda

~/anaconda3/bin/conda

$ which pip ~/anaconda3/envs/my_pymc_env/bin/pip

Now, I reinstalled loompy using: $ pip install -U loompy

I ran velocyto again... but, still the same issue...

2019-01-22 15:18:28,447 - DEBUG - Counting done! 2019-01-22 15:18:28,646 - DEBUG - Example of barcode: TGTCCCATCCATGCTC and cell_id: neurons_900_possorted_genome_bam_9CQVD:TGTCCCATCCATGCTC 2019-01-22 15:18:28,649 - DEBUG - Generating output file /ifs/scratch/.../rna-velocity/mouse_10x_website/velocyto/neurons_900_possorted_genome_bam_9CQVD.loom 2019-01-22 15:18:28,649 - DEBUG - Collecting row attributes 2019-01-22 15:18:28,868 - DEBUG - Generating data table 2019-01-22 15:18:29,496 - DEBUG - Writing loom file Traceback (most recent call last): File "/ifs/scratch/.../rna-velocity/velocyto.py-master/velocyto/commands/_run.py", line 286, in _run ds = loompy.create(filename=outfile, matrix=total, row_attrs=ra, col_attrs=ca, dtype="float32") TypeError: create() got an unexpected keyword argument 'matrix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/ifs/home/.../anaconda3/envs/my_pymc_env/bin/velocyto", line 11, in load_entry_point('velocyto', 'console_scripts', 'velocyto')() File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/ifs/scratch/.../rna-velocity/velocyto.py-master/velocyto/commands/run.py", line 113, in run samtools_memory=samtools_memory, dump=dump, verbose=verbose, additional_ca=additional_ca) File "/ifs/scratch/c2b2/ac_lab/st3179/rna-velocity/velocyto.py-master/velocyto/commands/_run.py", line 297, in _run loompy.create(filename=outfile, layers=tmp_layers, row_attrs=ra, col_attrs=ca, file_attrs={"velocyto.version": vcy.version, "velocyto.logic": logic}) File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/loompy/loompy.py", line 856, in create f = h5py.File(name=filename, mode='w') File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in init swmr=swmr) File "/ifs/home/.../anaconda3/envs/my_pymc_env/lib/python3.7/site-packages/h5py/_hl/files.py", line 176, in make_fid fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 105, in h5py.h5f.create OSError: Unable to create file (unable to lock file, errno = 37, error message = 'No locks available')

Where do you think is the problem? Many thanks!

slinnarsson commented 5 years ago

Hm. This one looks like a file system issue? Are you running on a server in an NFS-mounted directory? Then maybe there’s an issue with locking files: https://github.com/h5py/h5py/issues/1082

It might be worth trying the suggested environment variable.

somnathtagore commented 5 years ago

Hi,

I did 'export HDF5_USE_FILE_LOCKING=FALSE' and it worked ! Now, i get the .loom file perfectly.

Thanks!

shaaaarpy commented 4 years ago

Hi all, Is it necessary to have multiple files in the cellranger output or just bam file will work? Thank you

AAA-3 commented 3 years ago

The gtf file is not formatted in an adequate way (it is missing transcript id) and is therefore incompatible with velocyto... have you checked carefully this section of the tutorial: http://velocyto.org/velocyto.py/tutorial/cli.html#requirements-on-the-input-files Let me know if after fixing the gtf the problem persists.

Hello! Sorry for opening up an old issue, ut I had a simillar issue and wanted to know how to go about formatting the GTF gfile - this is the GTF file I sourced from Cell Ranger (10X reference version 2.1.0, GRCm38, Ensembl 84), which I assumed would have the required format...

Also, since my sequencing included the Cell Ranger pipeline, I am not sure why I get an error requesting I download the Sammtools package, seeing as cellranger generated bamfiles are already sorted...

Thoughts?

CODE velocyto run //home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/possorted_genome_bam.bam /home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/genes.gtf -b /home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/barcodes.tsv -o //home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/b2het_out -m /home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/mm10_repeat_mask.gtf

ERROR:

2021-07-26 16:49:08,244 - INFO - No SAMPLEID specified, the sample will be called possorted_genome_bam_HG0M5 (last 5 digits are a random-id to avoid overwriting some other file by mistake)
2021-07-26 16:49:08,244 - DEBUG - Using logic: Default
2021-07-26 16:49:08,247 - INFO - Read 3281 cell barcodes from /home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/barcodes.tsv
2021-07-26 16:49:08,247 - DEBUG - Example of barcode: AAACCTGAGGCTATCT and cell_id: possorted_genome_bam_HG0M5:AAACCTGAGGCTATCT-1
2021-07-26 16:49:08,274 - DEBUG - Peeking into /home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/possorted_genome_bam.bam
[E::idx_find_and_load] Could not retrieve index file for '/home/ali/Dokumente/RPractise/Run_alle_features/Velocity/PythonCodes/B2_HET/possorted_genome_bam.bam'
2021-07-26 16:49:08,276 - WARNING - Not found cell and umi barcode in entry 12 of the bam file
2021-07-26 16:49:08,276 - WARNING - Not found cell and umi barcode in entry 19 of the bam file
2021-07-26 16:49:08,276 - WARNING - Not found cell and umi barcode in entry 23 of the bam file
2021-07-26 16:49:08,276 - WARNING - Not found cell and umi barcode in entry 25 of the bam file
2021-07-26 16:49:08,276 - WARNING - Not found cell and umi barcode in entry 82 of the bam file
2021-07-26 16:49:08,277 - WARNING - Not found cell and umi barcode in entry 137 of the bam file
2021-07-26 16:49:08,277 - WARNING - Not found cell and umi barcode in entry 138 of the bam file
2021-07-26 16:49:08,278 - WARNING - Not found cell and umi barcode in entry 218 of the bam file
2021-07-26 16:49:08,278 - WARNING - Not found cell and umi barcode in entry 280 of the bam file
2021-07-26 16:49:08,278 - WARNING - Not found cell and umi barcode in entry 281 of the bam file
2021-07-26 16:49:08,278 - WARNING - Not found cell and umi barcode in entry 282 of the bam file
2021-07-26 16:49:08,278 - WARNING - Not found cell and umi barcode in entry 283 of the bam file
2021-07-26 16:49:08,279 - WARNING - Not found cell and umi barcode in entry 349 of the bam file
2021-07-26 16:49:08,280 - WARNING - Not found cell and umi barcode in entry 558 of the bam file
2021-07-26 16:49:08,280 - WARNING - Not found cell and umi barcode in entry 564 of the bam file
2021-07-26 16:49:08,281 - WARNING - Not found cell and umi barcode in entry 649 of the bam file
2021-07-26 16:49:08,281 - WARNING - Not found cell and umi barcode in entry 654 of the bam file
2021-07-26 16:49:08,282 - WARNING - Not found cell and umi barcode in entry 696 of the bam file
2021-07-26 16:49:08,282 - WARNING - Not found cell and umi barcode in entry 697 of the bam file
2021-07-26 16:49:08,283 - WARNING - Not found cell and umi barcode in entry 796 of the bam file
2021-07-26 16:49:08,283 - WARNING - Not found cell and umi barcode in entry 818 of the bam file
2021-07-26 16:49:08,283 - WARNING - Not found cell and umi barcode in entry 819 of the bam file
2021-07-26 16:49:08,283 - WARNING - Not found cell and umi barcode in entry 821 of the bam file
2021-07-26 16:49:08,283 - WARNING - Not found cell and umi barcode in entry 905 of the bam file
2021-07-26 16:49:08,283 - WARNING - Not found cell and umi barcode in entry 906 of the bam file
2021-07-26 16:49:08,283 - WARNING - Not found cell and umi barcode in entry 907 of the bam file
2021-07-26 16:49:08,284 - WARNING - Not found cell and umi barcode in entry 909 of the bam file
Traceback (most recent call last):
  File "/home/ali/anaconda3/bin/velocyto", line 8, in <module>
    sys.exit(cli())
  File "/home/ali/anaconda3/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/ali/anaconda3/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/ali/anaconda3/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ali/anaconda3/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ali/anaconda3/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/ali/anaconda3/lib/python3.8/site-packages/velocyto/commands/run.py", line 113, in run
    return _run(bamfile=bamfile, gtffile=gtffile, bcfile=bcfile, outputfolder=outputfolder,
  File "/home/ali/anaconda3/lib/python3.8/site-packages/velocyto/commands/_run.py", line 178, in _run
    sorting_process[ni] = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
  File "/home/ali/anaconda3/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/ali/anaconda3/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'samtools'
denvercal1234GitHub commented 2 years ago

@AAA-3 From issue #135 The data I am using is 10X data. I have now successfully made one loom file using the command run10x from count data generated from every version of cellranger (v1.3.1, v2.0.1, v3.0.0). I was able to create a loom file after I changed the following things:

-loading samtools along with velocyto.py -changing the name of the .bam file from samplename_possorted_genome_bam.bam to possorted_genome_bam.bam. -using my 10x cellranger count directory as the directory in my code. Previously, I was just directing the program to take the possorted_genome_bam.bam file.

Here is the code that has been successful for me:

module load python velocyto module load samtools cd name/data/counts/tissue/ velocyto run10x -m ~/name/analysis/velocyto/experiment/mm10_rmsk.gtf ./sampledirectory ~/refdata-cellranger-mm10-3.0.0/genes/genes.gtf

I make sure I have the velocyto and samtools modules loaded. I set my working directory to the directory where my 10x cellranger count output directory is. Then I run the velocyto run10x command, directing it to my repeat masker file, my genes file, and my 10x cellranger count output directory -> ./sampledirectory.