Closed LysSanzMoreta closed 6 months ago
@LysSanzMoreta Could you clarify whether the Versions you posted are for local or the docker env? I can reproduce this locally, actually, on my Mac using the main
branch now. I am interested how this is working for you locally. From what I can see, the dtype
that should have mapping between column of the dataframe and dtype simply does not have it.
-----
IPython 8.18.1
anndata 0.11.0.dev74+g71ed758.d20240220
session_info 1.0.0
-----
asciitree NA
asttokens NA
attr 23.1.0
awkward 2.5.1
awkward_cpp NA
cloudpickle 3.0.0
cython_runtime NA
dask 2023.12.0
dateutil 2.8.2
decorator 5.1.1
executing 2.0.1
fasteners 0.19
h5py 3.10.0
importlib_metadata NA
jedi 0.19.1
jinja2 3.1.2
markupsafe 2.1.3
msgpack 1.0.7
natsort 8.4.0
numcodecs 0.12.1
numpy 1.26.2
packaging 23.2
pandas 2.1.4
parso 0.8.3
pexpect 4.9.0
prompt_toolkit 3.0.43
psutil 5.9.6
ptyprocess 0.7.0
pure_eval 0.2.2
pyarrow 14.0.1
pygments 2.17.2
pytz 2023.3.post1
rich NA
scipy 1.11.4
setuptools 69.0.3
setuptools_scm NA
sitecustomize NA
six 1.16.0
sphinxcontrib NA
stack_data 0.6.3
tblib 3.0.0
tlz 0.12.0
toolz 0.12.0
traitlets 5.14.0
wcwidth 0.2.12
yaml 6.0.1
zarr 2.16.1
zipp NA
-----
Python 3.11.6 (main, Nov 2 2023, 04:39:43) [Clang 14.0.3 (clang-1403.0.22.14.1)]
macOS-13.6.1-arm64-arm-64bit
-----
Session information updated at 2024-02-20 10:30
One thing I notice is that it seems this dataset from the google drive link has shape (91588, 1330)
and not (91588, 589)
as your "working example" shows. Using the dataset you linked to in the google drive link:
import h5py
f = h5py.File('single_cell_data.h5ad', 'r')
f['X']
# <HDF5 dataset "X": shape (91588, 1330), type "<f4">
So just some clarification on what is happening locally would be great. I wonder if the file you sent is somehow corrupted.
@ilan-gold Sorry about the dataset difference, I have 2 toy examples I was playing with, both had the same issue, but I have uploaded the one with 91588x589 datapoints as well.
I also thought the file could be corrupted, but I could not figure out how to check for that, specially when it was working in the local host machine. See the installation script below.
The .yaml file that I was using last week, was the same both for the docker and the host machine. I have no clue why it was only working in the host machine.
I gotta say that I am new to Docker and last week I managed to crash my computer due to been unaware of the need to remove "untagged" images as well ... ops, so I had to reinstall everything from scratch. I have also realized that I should not be so harsh on package version specification and I have softened the .yaml and Dockerfiles. The harsh version specification was making very hard for micromamba to resolve a working environment and everything was crashing/incompatible.
So on that note, I have re-installed everything as stated by the new guidelines below. This has led me to been able to read both in the Docker container and the host machine the file labelled as NEW(dataset0) and still NOT been able to read the OLD(dataset0b) in the Docker container environment or the host machine. Actually, I am more interested in reading the OLD(dataset0b) because it contains more genes, despite of the name "OLD" (old is just in terms of this github issue).
Definitely there could be some dataset corruption, but there might be some library incompatibility somewhere as well? It is interesting since both files are generated using the same script ... one just has more genes than the other one. With the previous environment I could not read any of them in the Docker container.
I have bypassed the issue by instead reading the OLD(dataset0b) using thesc.read_hdf
command. So this code works in every environment:
adata = sc.read_hdf(sc_path, key="X")
which makes me think the dataset is not corrupted .. or?
NEW INSTALLATION GUIDELINES
.yaml file
name: base
channels:
- conda-forge
- pytorch
- defaults
- nvidia
- bioconda
dependencies:
- pyopenssl
- python=3.9
- requests
- pip:
- -r requirements.txt
- numpy
- pytorch::pytorch=2.1=py3.9_cuda12.1*
- torchvision=>0.16
- torchaudio
- biopython
- h5py
- scipy
- scikit-learn
- pandas
- matplotlib
- hdf5plugin
- pyro-ppl
- scanpy
- dill
- umap-learn
- seaborn
- dataframe_image
- xmltodict
- nbconvert
- bioconda::cellxgene
- conda-forge::lightning
- numba
- conda-forge::scvi-tools==1.0.4
- conda-forge::jax[cuda12_pip]
- conda-forge::jaxlib
- conda-forge::huggingface_hub
- conda-forge::leidenalg
requirements.txt
gget
DockerFile
FROM mambaorg/micromamba:1.5-jammy-cuda-12.2.2 as micromamba39
COPY --chown=$MAMBA_USER:$MAMBA_USER env.yaml /tmp/env.yaml
COPY --chown=$MAMBA_USER:$MAMBA_USER requirements.txt /tmp/requirements.txt
USER root
RUN micromamba install -y -n base -f /tmp/env.yaml && \
micromamba clean --all --yes
CMD ["/usr/bin/which","python3"] -> "/usr/bin/python3"
CMD ["/usr/bin/python3","/opt/.pycharm_helpers/packaging_tool.py","list"]
CMD ["/usr/bin/python3","/opt/.pycharm_helpers/remote_sync.py","--state-file","/tmp/e06641bc-ea0b-4a32-a780-2ed112e01659/.state.json","/tmp/bacecef2-e56a-4f46-9a40-fc180ff6cdd3"]
For the host machine I install it like:
#!/bin/bash
micromamba env remove -n base
micromamba install -f env.yaml
Scanpy version
scanpy 1.9.8 pyhd8ed1ab_0 conda-forge
``
Could you run the following in both settings and tell me what is in f['obs'].dtype.descr
?
import h5py
f = h5py.File('single_cell_data.h5ad', 'r')
print(f['obs'].dtype.descr)
@ilan-gold Yes!
File OLD(dataset0b) (which can NOT be read withsc.read_h5ad
and YES with sc.read.hdf
)
Docker container print output:
[('', '|O')]
Local host machine:
[('', '|O')]
File NEW(dataset0): (which can be read withsc.read_h5ad
and sc.read.hdf
)
Docker container:
print(f['obs'].dtype.descr)
AttributeError: 'Group' object has no attribute 'dtype'
Local host machine:
print(f['obs'].dtype.descr)
AttributeError: 'Group' object has no attribute 'dtype'
So on the good side, with the new environment, it seems like the Docker container and the local environment behave similarly.
I hope the print output sparks an inspiration on what is going on :)
@LysSanzMoreta You said you
still [had] NOT been able to read the OLD(dataset0b) in the Docker container environment or the host machine
which means the environments are behaving consistently now. Is that right? So this is different than when you reported the issue when they were acting differently, no?
What this little experiment highlights to me is that I do think the file is corrupted especially since the behavior is now identical across both. f['obs'].dtype.descr
was supposed to contain column names for the obs
dataframe, but as we see, it has none in either environment. And the reason this experiment doesn't work on the new file is that the way we write dataframes has changed since OLD was written. I would guess that OLD actually is old (or it was produced with a very old environment).
@ilan-gold Yeah, I would say the environments are working consistently now... (I never put my hand on fire for this kind of affirmations hehhe)
Ok ... interesting. I made both dataframes recently (2023-2024) (first NEW(dataset0) and then OLD(dataset0b)) so I would be surprised about that. However, I guess it is good that we can pin point to the file been corrupted somehow... I will perhaps review the pipeline then and see what I can do...although with the sc.read.hdf
I have been able to, for now bypass this issue.
If you have other ideas of what could it be, let me know ... I will investigate the obs
object, since you seem to point it as the corrupted part.
Thanks for your time!
Hey @LysSanzMoreta, I was talking to Ilan about this and it sounds like this OLD(dataset0b) was written by an old version of anndata, where I may be more help. Looking at this file now, I think this version of anndata may even pre-date me 😅.
Do you have the environment that created this file? I'd be curious to see what version of anndata wrote it.
cc: @flying-sheep
@ivirshup Well, let me see
The way that I am creating the files is by appending a bunch of gget adata files to each other like this:
max_genes = len(genes_list)
filed_dataframe=False
for genes in genes_sets:
adata = gget.cellxgene(....)
X_df = adata.X
.... some processing to remove duplicated genes etc....
if not filled_dataframe: #first time appending to the dataframe
print("First time appending to the hdf5 file")
all_tissues_storage.create_dataset('X',
data=X_df.to_numpy(),
compression="gzip",
chunks=True,
maxshape=(None, max_genes))
# .... append other keys similarly-----
else:
n_x = X_df.shape[0]
all_tissues_storage['X'].resize((all_tissues_storage['X'].shape[0] + n_x), axis=0)
all_tissues_storage['X'][-n_x:] = X_df.to_numpy()
#---append other keys similarly ------
An example of the gget adata file is: https://drive.google.com/file/d/1888V6da32RKRhG4vzj0B1eQ7oGfXTZty/view?usp=drive_link
The downloaded gget adata files (I have checked a few of them) respond like this:
i) Cannot be read with sc.read_hdf()
adata = sc.read_hdf(sc_path, key="X")
File "/home/lys/micromamba/lib/python3.9/site-packages/anndata/_io/read.py", line 130, in read_hdf
X = f[key][()]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/lys/micromamba/lib/python3.9/site-packages/h5py/_hl/group.py", line 359, in __getitem__
raise TypeError("Accessing a group is done with bytes or str, "
TypeError: Accessing a group is done with bytes or str, not <class 'tuple'>
ii) Can be read with sc.read_h5ad()
iii) Do not have the dtype attribute after using the _f = h5py.File(scpath, 'r') command
print(f['obs'].dtype.descr)
AttributeError: 'Group' object has no attribute 'dtype'
My current exact environment is:
Micromamba decided to install anndata 0.10.3 instead of 0.10.5, hd5py seems up to date. Perhaps the gget file was created with a different anndata version? I have no control over that ....
Since I decompose the dataframes in the original gget adata file to be able to concatenate them into a bigger file, I would think that would help with the fact that the gget adata file might have been created with an older version. However, there might be some issues within some of the components of the gget adata file.
Thanks again for your interest and time!
First off, I think you may want to use: anndata.experimental.concat_on_disk
for this task.
If you are creating a file manually, then you need to make sure you are creating something that adheres to the specification for the on disk format.
The downloaded gget adata files (I have checked a few of them) respond like this:
i) Cannot be read with sc.read_hdf() ii) Can be read with sc.read_h5ad()
This seems expected to me, as they are recent anndata files.
I would note that read_hdf
and read_h5ad
are different and have different expectations of the data.
iii) Do not have the dtype attribute after using the f = h5py.File(sc_path, 'r') command
print(f['obs'].dtype.descr)
I would only expect this for fairly old anndata files, from around when I started contributing (e.g. 5-ish years ago).
What is interesting about the file you shared is that it has the following keys:
$ h5ls -r single_cell_data_dataset0b.h5ad
/ Group
/X Dataset {91588/Inf, 1330/1389}
/col_names Dataset {1330}
/obs Dataset {91588/Inf, 10}
/obs_names Dataset {10}
/row_names Dataset {91588/Inf}
/var Dataset {4125/Inf, 4}
/var_names Dataset {4}
I don't really know what to expect here, since I've never seen a file like this (though there is some old code that uses these key names). I'm very curious what you called to write this.
Your code snippet also doesn't seem to add these keys, so I'm wondering where they came from.
@ivirshup Thanks for the suggestions :). Yeah, I definitely need to do more research on handling these large anndata files ... the anndata.experimental.concat_on_disk
is new? (since it has the experimental tag) I can have a look at it.
I am new to big anndata files (and docker hehhe)
Sorry about the confusion, to keep it short I did not write the entire concatenating loop and I skipped the lines where I also add the col_names
, row_names
etc keys. I do exactly the same as for the X
key :). I have added a comment
Regarding the naming of the keys, I might have followed some tutorial somewhere heheh
the anndata.experimental.concat_on_disk is new? (since it has the experimental tag) I can have a look at it.
Kinda new! From 0.10 on.
Dask also works well with dense chunks so that could be used here as well.
I might have followed some tutorial somewhere heheh
Do you know where this is from? I'm hoping not us...
Oh, ok thanks!
hahaha, probably some stackoverflow or so, it could have been old though hehhe
Alright, I'm going to close this as I don't think there is any required action on our side. Let me know if you don't think that is the case
Please make sure these conditions are met
Report
Hi! I am trying to coordinate several things to set up a Docker image that runs as an interpreter for Pycharm. Needless to say that in the Docker image, anndata 0.10.5.post1 is installed (via scvi-tools & scverse) and its is behaving weirdly.
I provide the Dockerfile and env.yaml information. The anndata version and dependencies are specified below.
Dockerfile:
env.yaml
The "problematic" code ... which is just reading an h5ad file that you can download from here:
OLD (dataset0b): https://drive.google.com/file/d/1NGo4TEu0WizC3xSta-zaumi6n5eUdOMF/view?usp=sharing
NEW (dataset0): https://drive.google.com/file/d/1uSKeuWi7_CK8-jOUIqX9YrFA_9_MzYp0/view?usp=sharing
Using the Docker image either as the interpreter in Pycharm or by running the image (docker run -it --gpus all -v "source:target:Z" bash) generates the following ERROR:
However when I install the same environment locally in my host machine, I can read the file perfectly fine. The output is:
At the beginning I thought that the error was due to a lack of read/write/execution permissions between the host and the docker container mount, however that does not seem the case. The error is that the dtype.fields becomes None in the original scanpy function __decode_structuredarray
I am not sure where the dtype is specified prior to this function (since dtype is not None when it reaches the _decode_structured_array function) and why the read behavior is different in the host machine versus the docker container. Note that the arr.dtype seems correct but it is ignored.
Perhaps you have an insight on what could be going on?
Thank you in advance. Let me know if you require anything else
Versions