search

mojaveazure / seurat-disk

Interfaces for HDF5-based Single Cell File Formats
https://mojaveazure.github.io/seurat-disk
GNU General Public License v3.0
142 stars 48 forks source link

Convert Error in dfile$obj_copy_from #30

Open igordot opened 3 years ago

igordot commented 3 years ago

I am trying to convert from AnnData to Seurat via h5Seurat. The process started, but then I got an error and an incomplete h5seurat was generated.

> Convert("file.h5ad", dest = "h5seurat", overwrite = TRUE) 
Warning: Unknown file type: h5ad
Warning: 'assay' not set, setting to 'RNA'
Creating h5Seurat file for version 3.1.5.9900
Adding X as data
Adding X as counts
Adding meta.features from var
Error in dfile$obj_copy_from(src_loc = source, src_name = "obs", dst_name = "meta.data") : 
  HDF5-API Errors:
    error #000: H5Ocopy.c in H5Ocopy(): line 233: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

    error #001: H5Ocopy.c in H5O__copy(): line 317: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

There were a few other error messages. I am just showing the first ones. Any idea how I may resolve this?

hayfre commented 3 years ago

Hi, I am getting the exact same error message when trying to convert from AnnData to Seurat. Any follow-up solutions?

diazdc commented 3 years ago

I'm dealing with the same problem. I thought it was a file permissions error at first, but chmod 777 didn't fix it.

selifeski commented 3 years ago

I am facing the same problem. Very weird because today it worked for another anndata but now I cannot make it worked and it gave the same long error.

billadelonge commented 3 years ago

Getting the same! :(

jayypaul commented 3 years ago

Same!!!

makloufg commented 3 years ago

Same here!

minhtran1309 commented 3 years ago

I am getting the same issue It has been already two months since the first time it gets reported. Has anyone solved it? Thanks

willrosenow commented 3 years ago

I am having the same issue

jloske commented 3 years ago

I also got the same error. I created a new env and reinstalled the packages, this time convert worked but when I wanted to use <- LoadH5Seurat() it ran into several problems. I even tried to remove the coloumns causing the issue but it kept running into next errors. Any solutions?

Validating h5Seurat file

Initializing RNA with data

Adding counts for RNA

Adding scale.data for RNA

Adding feature-level metadata for RNA

Error in `[[<-`(`*tmp*`, col.name, value = metadata): [[<- defined for objects of type "S4" only for subclasses of environment
Traceback:

1. LoadH5Seurat("file.h5seurat")
2. LoadH5Seurat.character("file.h5seurat")
3. LoadH5Seurat(file = hfile, assays = assays, reductions = reductions, 
 .     graphs = graphs, neighbors = neighbors, images = images, 
 .     meta.data = meta.data, commands = commands, misc = misc, 
 .     tools = tools, verbose = verbose, ...)
4. LoadH5Seurat.h5Seurat(file = hfile, assays = assays, reductions = reductions, 
 .     graphs = graphs, neighbors = neighbors, images = images, 
 .     meta.data = meta.data, commands = commands, misc = misc, 
 .     tools = tools, verbose = verbose, ...)
5. as.Seurat(x = file, assays = assays, reductions = reductions, 
 .     graphs = graphs, neighbors = neighbors, images = images, 
 .     meta.data = meta.data, commands = commands, misc = misc, 
 .     tools = tools, verbose = verbose, ...)
6. as.Seurat.h5Seurat(x = file, assays = assays, reductions = reductions, 
 .     graphs = graphs, neighbors = neighbors, images = images, 
 .     meta.data = meta.data, commands = commands, misc = misc, 
 .     tools = tools, verbose = verbose, ...)
7. AssembleAssay(assay = assay, file = x, slots = assays[[assay]], 
 .     verbose = verbose)
8. AddMetaData(object = obj, metadata = meta.data)
9. AddMetaData.Assay(object = obj, metadata = meta.data)
10. .AddMetaData(object = object, metadata = metadata, col.name = col.name)
jayypaul commented 3 years ago

Hi all,

I tried looking at my past notes where I got this to work but couldn't find exactly what I did. I remember that it had to do with the metadata sheet, something was wrong with the column names i think.... I suggest starting there! Best o luck

pcm32 commented 3 years ago

Also getting this. Using the pbmc3k_final.h5ad from the vignettes. Installed seurat with conda and seurat disk with an in-progress PR to conda forge:

hdf5                      1.10.6          nompi_h34ad4e8_1111    conda-forge
...
r-hdf5r                   1.3.3             r40h4a68bb8_0    conda-forge
...
r-seurat                  4.0.0             r40h4efb63e_0    conda-forge
r-seuratdisk              0.0.9015                      0    local
r-seuratobject            4.0.0             r40h4efb63e_0    conda-forge

Installation of r-seuratdisk happens through R CMD INSTALL --build . on commit 3848eab6fcb9ead489cabdad464921e43561e28c (see https://github.com/conda-forge/staged-recipes/pull/14134).

Any known issues with specific hdf5 library versions that we should avoid? The same SeuratDisk seems to work otherwise (other calls).

pcm32 commented 3 years ago

Also happens if I try building with latest commit (743d7397bd48c03575522881d3cded4796ab7006) or when trying with 0.09014 (95d27f636e719cf5fa49b02b05876354343138cd) as suggested in another issue.

pcm32 commented 3 years ago

Downgrading r-hdf5r to 1.3.2 seems to have no effect either.

andrewjkwok commented 3 years ago

Was wondering if there are any updates to this as I am getting the exact same problem?

chris-rands commented 3 years ago

Did anyone resolve this? There seem to be multiple related issues here: https://github.com/mojaveazure/seurat-disk/issues/47 https://github.com/mojaveazure/seurat-disk/issues/53 https://github.com/mojaveazure/seurat-disk/issues/64 https://github.com/mojaveazure/seurat-disk/issues/70 and on hdf5r
, https://github.com/hhoeflin/hdf5r/issues/166, and seurat https://github.com/satijalab/seurat/issues/3164

I have this error even when saving the h5ad object with compression="gzip" or "lzf" Would be grateful for any help cc @mojaveazure

chris-rands commented 3 years ago

I think I found a (partial) solution, save with gzip compression and as dense matrix, e.g. python -c "import scanpy as sc; adata = sc.read('pbmc3k_final.h5ad'); adata.write('pbmc3k_final_gzip.h5ad', compression='gzip', force_dense=True)" .

I am also having to change some meta-data attributes, specifically adding adata.raw and deleting the adata.var attributes

Echo226 commented 3 years ago

I am getting the same issue.

flde commented 3 years ago

Getting the same error.

zhanglab2008 commented 3 years ago

same issue for me

hayfre commented 3 years ago

To avoid this error you can convert from AnnData to loom and then use the loomR package to convert from loom to Seurat. https://satijalab.org/loomr/loomr_tutorial

MikaQiao commented 3 years ago

I am getting the same issue.

Xenolevis commented 3 years ago

I fixed this by removing libhdf5-* version 1.10.x that came with my system (linux) and compiling my own hdf5 API. Latest 1.8 version (1.8.22) seems to work fine. I have also tried 1.10.3 but no luck. Earlier 1.10.x versions are not available from the HDF Group.

ACastanza commented 3 years ago

I am also experiencing this issue.

Full Error Message:

Convert(infile, dest = "h5seurat", overwrite = TRUE)
Warning: Unknown file type: h5ad
Warning: 'assay' not set, setting to 'RNA'
Creating h5Seurat file for version 3.1.5.9900
Adding X as data
Adding X as counts
Adding meta.features from var
Error in dfile$obj_copy_from(src_loc = source, src_name = "obs", dst_name = "meta.data") :
  HDF5-API Errors:
    error #000: H5Ocopy.c in H5Ocopy(): line 233: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

    error #001: H5Ocopy.c in H5O__copy(): line 317: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

    error #002: H5Ocopy.c in H5O__copy_obj(): line 1221: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

    error #003: H5Ocopy.c in H5O__copy_header(): line 1165: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

    error #004: H5Ocopy.c in H5O__copy_header_real(): line 872: unable to perform 'post copy' operation on message
        class: HDF5
        major: Object header
        minor: Unable to initialize object

    error #005: H5Ostab.c in H5O__stab_post_copy_file(): line 403: iteration operator failed
        class: HD
Error in private$closeFun(id) : HDF5-API Errors:
    error #000: H5F.c in H5Fclose(): line 675: closing file ID failed
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #001: H5Fint.c in H5F__close(): line 1952: decrementing file ID failed
        class: HDF5
        major: File accessibilty
        minor: Unable to decrement reference count

    error #002: H5I.c in H5I_dec_app_ref(): line 1300: can't decrement ID ref count
        class: HDF5
        major: Object atom
        minor: Unable to decrement reference count

    error #003: H5Fint.c in H5F__close_cb(): line 2010: can't close file
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #004: H5Fint.c in H5F_try_close(): line 2181: problems closing file
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #005: H5Fint.c in H5F__dest(): line 1277: problems closing file
        class: HDF5
        major: File
>

h5ad file was written out from scVelo using AnnData v0.7.6 with compression="gzip". Also tried with force_dense=True and this did not resolve the issue.

Failing h5ad file here: https://drive.google.com/file/d/1Mt6-yiGyWCLC5pwc5AmVMRlA7rbQps1K/view?usp=sharing

Xenolevis commented 3 years ago

@ACastanza I'm able to convert your file on my setup with hdf5 API 1.8.22. What OS are you on?

ACastanza commented 3 years ago

@Xenolevis I am on a Mac (10.15.7), with packages installed via conda.

I'm not entirely sure which package would be providing the HDF5 API you mention, but the HDF5 packages installed on my system are as follows:

bioconductor-hdf5array 1.18.1 r40h68a2ddb_0 bioconda bioconductor-rhdf5 2.34.0 r40h832b226_1 bioconda bioconductor-rhdf5filters 1.2.0 r40h64ad5ed_0 bioconda bioconductor-rhdf5lib 1.12.1 r40h68a2ddb_0 bioconda hdf5 1.10.6 nompi_h34ad4e8_1111 conda-forge r-hdf5r 1.3.3 r40h4a68bb8_0 conda-forge

Xenolevis commented 3 years ago

@ACastanza it's the hdf5 one. None of the available 1.10.x. or 1.12.x that I have tested work for me. I see 1.8.20 is available from conda-forge, and version 1.8.22 from Homebrew. Unfortunately I don't quite speak macOS, so I can't offer better directions. Basically you would have to:

  1. remove the R packages Seurat-disk and hdf5r.
  2. downgrade the hdf5 library.
  3. reinstall the R packages.

Good luck!

ACastanza commented 3 years ago

I see, I'll give it a shot. Or just create a new conda environment with that version.

It does sound like this is a bug in seurat-disk then; that it needs to be updated to support some change in the api, and until then, the package manifests need to explicitly specify hdf5<.10.6 so that conda can install a working version when it resolves the dependency conflicts.

Xenolevis commented 3 years ago

At least equally likely to be an issue of hdf5r. It's known to be compatible only with version up to 1.10.0 of the hdf5 api, according to its readme.

ACastanza commented 3 years ago

Yeah that seems to be a problem with hdf5r as well, in it's package description it's specifying

SystemRequirements: HDF5 (>= 1.8.13) https://github.com/hhoeflin/hdf5r/blob/master/DESCRIPTION

which is allowing it to install the incompatible 1.10.x versions.

So something definitely changed in the latest releases that these packages need to account for.

ACastanza commented 3 years ago

Even a very basic anaconda environment can't seem to resolve the dependencies necessary to get this older version of hdf5 to install. This is definitely something that the seurat-disk and hdf5r folks are going to need to resolve.

Edit: the fundamental incompatibiltiy seems to be down to the libfortran version required, it's impossible to install R 4.x alongside this old version of the hdf5 library:

r-base=4 -> libgfortran[version='5.*|>=4.0.0,<5.0.0.a0'] hdf5=1.8.20 -> libgfortran[version='>=3.0.1,<4.0.0.a0']

Xenolevis commented 3 years ago

Could try to install a system-wide 1.8.22 hdf5 library with Homebrew (it has no Fortran dependencies), then use remotes::install_github within R to get the hdf5r and SeuratDisk packages from their repos.

teresho4 commented 2 years ago

I got the same problem, I cleaned metadata from anndata.obs and for me, it works. It seems that seurat does not like some symbols in the name of columns or can't add all the metadata to seurat object.

wbvguo commented 2 years ago

I got the same problem, have anyone figured out a solution?

teresho4 commented 2 years ago

I got the same problem, have anyone figured out a solution?

Did you check column names in metadata? Try not to use specific symbols as '-'. You can try to convert without metadata at first.

ZeitgeberH commented 2 years ago

I found that If the anndata object is transposed before saving into .h5ad, then "Convert" function proceeds without the above errors. However, this obviously swaps 'genes' with 'sample'. Not sure why though.

manolisathan commented 2 years ago

conda create -n hdf5_1820 r-base=4.0.1 hdf5=1.8.20 conda activate hdf5_1820 R install.packages("remotes") remotes::install_github("mojaveazure/seurat-disk")

and then conversion worked properly!

bsierieb1 commented 2 years ago

afters some modifications, @manolisathan's solution worked for me! conda found conflicts between hdf5 1.8.20 and R 4.0.1, so i ended up installing an older version of R and also older versions of Seurat and other packages:

conda create -n hdf5_1820 r-base=3.6.0 hdf5=1.8.20
conda install -c conda-forge libgit2 # required by devtools
conda activate hdf5_1820
R
install.packages('devtools')
install.packages('remotes')
devtools::install_version(package='irlba', version=package_version('2.3.3')) # required by Seurat
devtools::install_version(package='rsvd', version=package_version('1.0.3')) # required by Seurat
install.packages('https://cran.r-project.org/src/contrib/Archive/spatstat/spatstat_1.64-1.tar.gz', repos=NULL, type="source") # required by Seurat
devtools::install_version(package='Seurat', version=package_version('3.2.3'))
devtools::install_version(package='SeuratObject', version=package_version('4.0.0'))
remotes::install_github('mojaveazure/seurat-disk')
BingjieZhang commented 2 years ago

An easy fix for me is to dump all the dense scaled and corrected matrix in python and then do the conversion. 

import scanpy as sc
import pandas as pd
import anndata as ad
import os
import numpy as np

adata_all = sc.read_h5ad('scRNA_processed_cleaned.h5ad')
adata_all.raw.to_adata().write('scRNA_raw_cleaned.h5ad')

And then use scRNA_raw_cleaned.h5ad as the input.

gly228003 commented 2 years ago

@bsierieb1 I used your script but it failed at the step install.packages('devtools') in R 3.6 environment. Then I used conda to install devtools conda install r-devtools which took too long time to complete the installation. Could you offer a solution? thanks!

mihem commented 1 year ago

The following worked for me: Use anndata2rihttps://github.com/theislab/anndata2ri to convert AnnData to a SingleCellExperiment object. Attention: You need to install r2py before that, so e.g. in Ubuntu sudo apt install python3-rpy2

Use a Jupyter Notebook and follow the tutorial, so:

import scanpy
import anndata2ri
anndata2ri.activate()
%load_ext rpy2.ipython
adata = scanpy.read_h5ad("refdata.h5ad")

%%R -i adata
saveRDS(adata, "adata_sce.rds")

Convert SCE to Seurat

seu <- as.Seurat(adata_sce, counts = "counts", data = NULL)

Note: I needed to specify data as NULL because there were no normalized data in this object.

AnjaliS1 commented 1 year ago

Hi same issue. Has this been resolved?

BirkirRey commented 10 months ago

I got the same problem, I cleaned metadata from anndata.obs and for me, it works. It seems that seurat does not like some symbols in the name of columns or can't add all the metadata to seurat object.

This! Before writing anndata to disk, make sure all column names in .obs and .var only have alphanumeric values(no spaces and symbols). Also, be sure that the anndata.X are raw counts, not normalized values.

leahrosen commented 8 months ago

In case anyone comes across this in future (potentially me) when converting h5Seurat to h5ad, just wanted to mention that I got a similar issue (I got an hdf5-api error "unable to synchronously copy object"). Turns out the issue was using a Seurat v5 assay. For more details see: https://satijalab.org/seurat/articles/seurat5_essential_commands But this was the code that fixed it for me (potentially updating something would help, but this was quick and worked 👍 ).

srat[["RNA3"]] <- as(object = srat[["RNA"]], Class = "Assay")
DefaultAssay(srat) <- "RNA3"
srat[["RNA"]] <- NULL
srat[["RNA"]] <- srat[["RNA3"]]
DefaultAssay(srat) <- "RNA"
srat[["RNA3"]] <- NULL
Violet969 commented 7 months ago

Same problem, but I tried to degrade the Seurat version from V5 to V4.4, the error is disappeared.

b-coli commented 6 months ago

In case anyone comes across this in future (potentially me) when converting h5Seurat to h5ad, just wanted to mention that I got a similar issue (I got an hdf5-api error "unable to synchronously copy object"). Turns out the issue was using a Seurat v5 assay. For more details see: https://satijalab.org/seurat/articles/seurat5_essential_commands But this was the code that fixed it for me (potentially updating something would help, but this was quick and worked 👍 ).

srat[["RNA3"]] <- as(object = srat[["RNA"]], Class = "Assay")
DefaultAssay(srat) <- "RNA3"
srat[["RNA"]] <- NULL
srat[["RNA"]] <- srat[["RNA3"]]
DefaultAssay(srat) <- "RNA"
srat[["RNA3"]] <- NULL

I had the same issue, and this fix worked for me