search

MarioniLab / DropletUtils

Clone of the Bioconductor repository for the DropletUtils package.
https://bioconductor.org/packages/devel/bioc/html/DropletUtils.html
54 stars 27 forks source link

swappedDrops "Missing filters: deflate" error? #30

Closed steelman82 closed 4 years ago

steelman82 commented 4 years ago

Hi:

I can't seem to get swappedDrops to run due to what appears to be some sort of file handling issue(?). Everything worked flawlessly when I followed the instructions to mock up the data and run swappedDrops, but every time I try to use "real" molecule_info.h5 files from 10X, I get the "Missing filters: deflate" error detailed below. I've tried using in-house generated molecule_info.h5 files (processed via Cell Ranger 3.1.0) as well as ones downloaded from 10X (also from Cell Ranger 3.1.0).

I noticed that a previously reported issue described a similar problem, but it sounded like the problem in that case was an incompatibility between an earlier version of DropletUtils (version 1.2.2) and Cell Ranger 3.1. The solution there appeared to be an update to DropletUtils version 1.3.10, but I'm assuming that the current version of DropletUtils (1.4.3) is the correct version to use?

Apologies in advance if I'm just being dense and missing something obvious. And many, many thanks in advance for any help/guidance you can offer.

Session output:

suppressMessages(library(DropletUtils)) Warning messages: 1: package ‘SingleCellExperiment’ was built under R version 3.6.1 2: package ‘BiocGenerics’ was built under R version 3.6.1 3: package ‘GenomeInfoDb’ was built under R version 3.6.1 4: package ‘Biobase’ was built under R version 3.6.1 5: package ‘DelayedArray’ was built under R version 3.6.1 6: package ‘matrixStats’ was built under R version 3.6.1 mol.info <- c("/home/ubuntu/test_swapped_drops/h5dir/135_molecule_info.h5","/home/ubuntu/test_swapped_drops/h5dir/136_molecule_info.h5") mol.info [1] "/home/ubuntu/test_swapped_drops/h5dir/135_molecule_info.h5" [2] "/home/ubuntu/test_swapped_drops/h5dir/136_molecule_info.h5" out <- swappedDrops(mol.info) Error: Unable to read dataset. Not all required filters available. Missing filters: deflate sessionInfo() R version 3.6.0 (2019-04-26) Platform: x86_64-conda_cos6-linux-gnu (64-bit) Running under: Ubuntu 18.04.2 LTS

Matrix products: default BLAS/LAPACK: /home/ubuntu/miniconda3/lib/R/lib/libRblas.so

locale: [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] parallel stats4 stats graphics grDevices utils datasets [8] methods base

other attached packages: [1] DropletUtils_1.4.3 SingleCellExperiment_1.6.0 [3] SummarizedExperiment_1.14.1 DelayedArray_0.10.0
[5] BiocParallel_1.18.1 matrixStats_0.55.0
[7] Biobase_2.44.0 GenomicRanges_1.36.1
[9] GenomeInfoDb_1.20.0 IRanges_2.18.3
[11] S4Vectors_0.22.1 BiocGenerics_0.30.0

loaded via a namespace (and not attached): [1] Rcpp_1.0.2 edgeR_3.26.8 XVector_0.24.0
[4] zlibbioc_1.30.0 lattice_0.20-38 tools_3.6.0
[7] grid_3.6.0 rhdf5_2.28.1 dqrng_0.2.1
[10] R.oo_1.23.0 HDF5Array_1.12.3 Matrix_1.2-17
[13] GenomeInfoDbData_1.2.2 Rhdf5lib_1.6.3 R.utils_2.9.0
[16] bitops_1.0-6 RCurl_1.95-4.12 limma_3.40.6
[19] compiler_3.6.0 R.methodsS3_1.7.1 locfit_1.5-9.1

LTLA commented 4 years ago

Can you give an example of a public dataset where this occurs?

steelman82 commented 4 years ago

WOW. Thank you for the amazingly quick response. I've tested it on a few datasets from 10X and encountered the same issue, which makes me think it must just be something silly on my end. In each case, I've been working with the *_molecule_info.h5 files (Per-molecule read information):

5k Peripheral blood mononuclear cells (PBMCs) from a healthy donor with cell surface proteins (Next GEM) - Cell Ranger 3.1.0 https://support.10xgenomics.com/single-cell-gene-expression/datasets/3.1.0/5k_pbmc_protein_v3_nextgem (basically I just duplicated the _molecule_info.h5 file to "simulate" multiple samples to do a test run)

and

1k PBMCs from a Healthy Donor (v3 chemistry) - Cell Ranger 3.0.0 https://support.10xgenomics.com/single-cell-gene-expression/datasets/3.0.0/pbmc_1k_v3

plus

1k Brain Cells from an E18 Mouse (v3 chemistry) - Cell Ranger 3.0.0 https://support.10xgenomics.com/single-cell-gene-expression/datasets/3.0.0/neuron_1k_v3

From: Aaron Lun notifications@github.com Reply-To: MarioniLab/DropletUtils reply@reply.github.com Date: Tuesday, November 12, 2019 at 4:22 PM To: MarioniLab/DropletUtils DropletUtils@noreply.github.com Cc: Scott Steelman ssteelman@cellarity.com, Author author@noreply.github.com Subject: Re: [MarioniLab/DropletUtils] swappedDrops "Missing filters: deflate" error? (#30)

Can you give an example of a public dataset where this occurs?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/MarioniLab/DropletUtils/issues/30?email_source=notifications&email_token=AMTA2OQUKXHR4Z3V6HYQLP3QTMNA3A5CNFSM4JMIRSXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED37GJQ#issuecomment-553120550, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMTA2OX66YSI2WMAA2IW2P3QTMNA3ANCNFSM4JMIRSXA.

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you should delete this message. You are hereby notified that any disclosure, copying, distribution or taking action in relation to the contents of this information is strictly prohibited and may be unlawful.

LTLA commented 4 years ago

Pulling down 5k_pbmc_protein_v3_nextgem, saving it as blah.h5 and blah2.h5 and running:

library(DropletUtils)
X <- swappedDrops(c("blah.h5", "blah2.h5"))

Works fine for me on R 3.6.1, DropletUtils 1.6.1. Best advice is to try updating to R 3.10, maybe it's already fixed.

PeteHaitch commented 4 years ago

I don't have any answers, but some suggestions that may be helpful. I think this is an issue with the installation of Rhdf5lib/rhdf5 on your machine, @steelman82.

I see you're using conda for some of your R installation (at least, BLAS/LAPACK) - I think there can sometimes be issues mixing and matching bioconda packages and packages installed using BiocManager::install().

I believe the error

Not all required filters available.
Missing filters: deflate

means that zlib isn't properly available for deflating gzipped HDF5 files (which is most of them, certainly the 10x HDF5 files, I believe). Some colleagues of mine were battling 'deflate'-related errors installing Rhdf5lib/rhdf5 on their server within a conda environment and it required some work to get the appropriate zlib headers to talk properly to the R installation.

Can you re-create this erorr just using, say, an example from the rhdf5 package? I don't think this is an error with DropletUtils per se.

steelman82 commented 4 years ago

You sir, are my HERO. I grabbed the public AMI for the latest Bioconductor release (3.10) and installed DropletUtils 1.6.1 and everything worked like a charm. Thank you SO, SO much for your help. And thank you for all the effort you've put into addressing the index swapping issue--it is greatly appreciated.

From: Aaron Lun notifications@github.com Reply-To: MarioniLab/DropletUtils reply@reply.github.com Date: Tuesday, November 12, 2019 at 4:51 PM To: MarioniLab/DropletUtils DropletUtils@noreply.github.com Cc: Scott Steelman ssteelman@cellarity.com, Author author@noreply.github.com Subject: Re: [MarioniLab/DropletUtils] swappedDrops "Missing filters: deflate" error? (#30)

Pulling down 5k_pbmc_protein_v3_nextgem, saving it as blah.h5 and blah2.h5 and running:

library(DropletUtils)

X <- swappedDrops(c("blah.h5", "blah2.h5"))

Works fine for me on R 3.6.1, DropletUtils 1.6.1. Best advice is to try updating to R 3.10, maybe it's already fixed.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/MarioniLab/DropletUtils/issues/30?email_source=notifications&email_token=AMTA2OXUKSZJE5MKR3EMDVLQTMQOTA5CNFSM4JMIRSXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED4CFKY#issuecomment-553132715, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMTA2OUCTEHYEXW4MJ3VNM3QTMQOTANCNFSM4JMIRSXA.

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you should delete this message. You are hereby notified that any disclosure, copying, distribution or taking action in relation to the contents of this information is strictly prohibited and may be unlawful.

LTLA commented 4 years ago

Thanks @PeteHaitch, closing.

steelman82 commented 4 years ago

Hi Peter:

Thank you for the quick response, but it turns out Aaron was even quicker. 😊

It was indeed an issue on my end and NOT an issue with DropletUtils at all. I updated to the latest version of Bioconductor and once I did that, everything worked PERFECTLY.

Thank you so much for taking the time to help me though what ended up being a very newb issue. I very much appreciate it.

From: Peter Hickey notifications@github.com Reply-To: MarioniLab/DropletUtils reply@reply.github.com Date: Tuesday, November 12, 2019 at 5:58 PM To: MarioniLab/DropletUtils DropletUtils@noreply.github.com Cc: Scott Steelman ssteelman@cellarity.com, Mention mention@noreply.github.com Subject: Re: [MarioniLab/DropletUtils] swappedDrops "Missing filters: deflate" error? (#30)

I don't have any answers, but some suggestions that may be helpful. I think this is an issue with the installation of Rhdf5lib/rhdf5 on your machine, @steelman82https://github.com/steelman82.

I see you're using conda for some of your R installation (at least, BLAS/LAPACK) - I think there can sometimes be issues mixing and matching bioconda packages and packages installed using BiocManager::install().

I believe the error

Not all required filters available.

Missing filters: deflate

means that zlib isn't properly available for deflating gzipped HDF5 files (which is most of them, certainly the 10x HDF5 files, I believe). Some colleagues of mine were battling 'deflate'-related errors installing Rhdf5lib/rhdf5 on their server within a conda environment and it required some work to get the appropriate zlib headers to talk properly to the R installation.

Can you re-create this erorr just using, say, an example from the rhdf5 package? I don't think this is an error with DropletUtils per se.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/MarioniLab/DropletUtils/issues/30?email_source=notifications&email_token=AMTA2OX4K6Y3W6Z6LB3HNKDQTMYKLA5CNFSM4JMIRSXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED4H5MA#issuecomment-553156272, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMTA2OUCNLWGRKXRXOKZW5LQTMYKLANCNFSM4JMIRSXA.

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you should delete this message. You are hereby notified that any disclosure, copying, distribution or taking action in relation to the contents of this information is strictly prohibited and may be unlawful.