search

Dana-Farber-AIOS / pathml

Tools for computational pathology
https://pathml.org
GNU General Public License v2.0
387 stars 81 forks source link

SlideData can't read .h5 file #159

Closed MohamedOmar2020 closed 2 years ago

MohamedOmar2020 commented 3 years ago

Hello, I have a problem with reading the slidedata object saved as .h5 file. This is the code used to write the file to disk (works fine):

slidedata.write('data/slidedata/tile1_processed.h5')

When I try to load it back using SlideData function:

slidedata_h5 = SlideData('data/slidedata/tile1_processed.h5')

I get this error:

Traceback (most recent call last): File "/Users/mohamedomar/opt/anaconda3/envs/pathml2/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in slidedata_h5 = SlideData('data/slidedata/sanjay1_tile1_processed.h5') File "/Users/mohamedomar/Documents/Research/Tutorials/pathml2/pathml/core/slide_data.py", line 193, in init self.h5manager = pathml.core.h5managers.h5pathManager(h5path=f) File "/Users/mohamedomar/Documents/Research/Tutorials/pathml2/pathml/core/h5managers.py", line 48, in init self.counts = readcounts(h5path["counts"]) File "/Users/mohamedomar/Documents/Research/Tutorials/pathml2/pathml/core/utils.py", line 117, in readcounts return anndata.read_h5ad(path.name) File "/Users/mohamedomar/opt/anaconda3/envs/pathml2/lib/python3.8/site-packages/anndata/_io/h5ad.py", line 408, in read_h5ad with h5py.File(filename, "r") as f: File "/Users/mohamedomar/opt/anaconda3/envs/pathml2/lib/python3.8/site-packages/h5py/_hl/files.py", line 424, in init fid = make_fid(name, mode, userblock_size, File "/Users/mohamedomar/opt/anaconda3/envs/pathml2/lib/python3.8/site-packages/h5py/_hl/files.py", line 190, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 96, in h5py.h5f.open OSError: Unable to open file (file signature not found)

Thanks in advance.

ryanccarelli commented 3 years ago

@MohamedOmar2020 This is using the latest pip version? I will try to reproduce

Update: was able to reproduce, will work on a fix

MohamedOmar2020 commented 3 years ago

Yep I am using the latest pip version. Another issue with that version, I think it is not compatible with h5py v3.1.0. Whenever I load the package, I get this error:

module 'h5py' has no attribute 'dataset'

This error disappears when I upgrade h5py to v3.3.0

ryanccarelli commented 3 years ago

I don't get the h5py error, could you 

conda env list

As an update I fixed this reading problem, but some of the slidedata.counts.obs won't read (like slice). Looking for a fix. You can see progress in PR #160

Thanks for your patience while we iron this out

MohamedOmar2020 commented 3 years ago

These are my conda envs (pathml2 is the one I am working with):

base /Users/mohamedomar/opt/anaconda3 conda /Users/mohamedomar/opt/anaconda3/envs/conda pathml /Users/mohamedomar/opt/anaconda3/envs/pathml pathml2 * /Users/mohamedomar/opt/anaconda3/envs/pathml2 pyscenic /Users/mohamedomar/opt/anaconda3/envs/pyscenic r-reticulate /Users/mohamedomar/opt/anaconda3/envs/r-reticulate /usr/local/anaconda3

These are the packages in pathml2:

packages in environment at /Users/mohamedomar/opt/anaconda3/envs/pathml2:

Name Version Build Channel alabaster 0.7.12 pypi_0 pypi anndata 0.7.6 pypi_0 pypi annoy 1.17.0 pypi_0 pypi appdirs 1.4.4 pyh9f0ad1d_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge babel 2.9.1 pypi_0 pypi backports-entry-points-selectable 1.1.0 pypi_0 pypi batchglm 0.7.4 pypi_0 pypi bbknn 1.5.1 pypi_0 pypi blosc 1.21.0 he49afe7_0 conda-forge bokeh 2.3.3 py38h50d1736_0 conda-forge boto3 1.17.86 pypi_0 pypi botocore 1.20.86 pypi_0 pypi brotli 1.0.9 h0d85af4_5 conda-forge brotli-bin 1.0.9 h0d85af4_5 conda-forge brotlipy 0.7.0 py38h96a0964_1001 conda-forge brunsli 0.1 h046ec9c_0 conda-forge bzip2 1.0.8 h0d85af4_4 conda-forge ca-certificates 2021.5.30 h033912b_0 conda-forge cachey 0.2.1 pypi_0 pypi cellpose 0.6.5 pypi_0 pypi certifi 2021.5.30 py38h50d1736_0 conda-forge cffi 1.14.6 py38h9688ba1_0 conda-forge cfgv 3.3.0 pypi_0 pypi chardet 4.0.0 py38h50d1736_1 conda-forge charls 2.2.0 h046ec9c_0 conda-forge charset-normalizer 2.0.0 pyhd8ed1ab_0 conda-forge click 8.0.1 py38h50d1736_0 conda-forge cloudpickle 1.6.0 py_0 conda-forge colorama 0.4.4 pypi_0 pypi commonmark 0.9.1 pypi_0 pypi cryptography 3.4.7 py38h1fa4640_0 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.11.0 py38h96a0964_3 conda-forge dask 2021.7.1 pypi_0 pypi dask-core 2021.5.1 pyhd8ed1ab_0 conda-forge decorator 5.0.9 pyhd8ed1ab_0 conda-forge deepcell 0.9.0 pypi_0 pypi diffxpy 0.7.4 pypi_0 pypi distlib 0.3.2 pypi_0 pypi distributed 2021.7.1 pypi_0 pypi docstring-parser 0.8.1 pypi_0 pypi docutils 0.16 pypi_0 pypi et-xmlfile 1.1.0 pypi_0 pypi fbpca 1.0 pypi_0 pypi filelock 3.0.12 pypi_0 pypi freetype 2.10.4 h4cff582_1 conda-forge freetype-py 2.2.0 pypi_0 pypi fsspec 2021.6.1 pypi_0 pypi future 0.18.2 py38h50d1736_3 conda-forge gast 0.3.3 pypi_0 pypi geosketch 1.2 pypi_0 pypi giflib 5.2.1 hbcb3906_2 conda-forge grpcio 1.32.0 pypi_0 pypi h5py 3.3.0 pypi_0 pypi harmonypy 0.0.5 pypi_0 pypi heapdict 1.0.1 py_0 conda-forge hsluv 5.0.2 pypi_0 pypi identify 2.2.12 pypi_0 pypi idna 3.1 pyhd3deb0d_0 conda-forge imagecodecs 2021.4.28 py38hd27bfcf_0 conda-forge imageio 2.9.0 py_0 conda-forge imagesize 1.2.0 pypi_0 pypi iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge intervaltree 2.1.0 pypi_0 pypi ipython 7.24.1 pypi_0 pypi javabridge 1.0.16 pypi_0 pypi jbig 2.1 h0d85af4_2003 conda-forge jinja2 3.0.1 pyhd8ed1ab_0 conda-forge jmespath 0.10.0 pypi_0 pypi jpeg 9d hbcb3906_0 conda-forge jupyter 1.0.0 pypi_0 pypi jupyter-contrib-core 0.3.3 pypi_0 pypi jxrlib 1.1 h35c211d_2 conda-forge keras-nightly 2.5.0.dev2021032900 pypi_0 pypi kiwisolver 1.3.1 py38h12bbefe_1 conda-forge lcms2 2.12 h577c468_0 conda-forge lerc 2.2.1 h046ec9c_0 conda-forge libaec 1.0.5 he49afe7_0 conda-forge libblas 3.9.0 8_mkl conda-forge libbrotlicommon 1.0.9 h0d85af4_5 conda-forge libbrotlidec 1.0.9 h0d85af4_5 conda-forge libbrotlienc 1.0.9 h0d85af4_5 conda-forge libcblas 3.9.0 8_mkl conda-forge libcxx 12.0.1 habf9029_0 conda-forge libdeflate 1.7 h35c211d_5 conda-forge libffi 3.3 h046ec9c_2 conda-forge libgfortran 5.0.0 9_3_0_h6c81a4c_23 conda-forge libgfortran5 9.3.0 h6c81a4c_23 conda-forge liblapack 3.9.0 8_mkl conda-forge libpng 1.6.37 h7cec526_2 conda-forge libprotobuf 3.15.8 hcf210ce_0 conda-forge libtiff 4.3.0 h1167814_1 conda-forge libwebp-base 1.2.0 h0d85af4_2 conda-forge libzopfli 1.0.3 h046ec9c_0 conda-forge llvm-openmp 12.0.1 hda6cdc1_1 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.3 he49afe7_1 conda-forge magicgui 0.2.10 pypi_0 pypi mahotas 1.4.11 pypi_0 pypi markupsafe 2.0.1 py38h96a0964_0 conda-forge matplotlib 3.1.3 pypi_0 pypi matplotlib-inline 0.1.2 pypi_0 pypi mkl 2020.4 h08c4f10_301 conda-forge more-itertools 8.8.0 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.2 py38h12bbefe_1 conda-forge napari 0.4.10 pypi_0 pypi napari-console 0.0.3 pypi_0 pypi napari-plugin-engine 0.1.9 pypi_0 pypi napari-svg 0.1.5 pypi_0 pypi nbconvert 6.0.7 pypi_0 pypi nbsphinx 0.8.5 pypi_0 pypi nbsphinx-link 1.3.0 pypi_0 pypi ncurses 6.2 h2e338ed_4 conda-forge networkx 2.5 py_0 conda-forge ninja 1.10.2 h9a9d8cb_0 conda-forge nodeenv 1.6.0 pypi_0 pypi numpy 1.18.5 pypi_0 pypi numpydoc 1.1.0 pypi_0 pypi olefile 0.46 pyh9f0ad1d_1 conda-forge opencv-contrib-python 4.5.3.56 pypi_0 pypi openjdk 8.0.152 h1de35cc_3 openjpeg 2.4.0 h6e7aa92_1 conda-forge openpyxl 3.0.7 pypi_0 pypi openslide-python 1.1.2 pypi_0 pypi openssl 1.1.1k h0d85af4_0 conda-forge packaging 21.0 pyhd8ed1ab_0 conda-forge pandas 1.1.5 pypi_0 pypi partd 1.2.0 pyhd8ed1ab_0 conda-forge pathml 1.0.0 pypi_0 pypi pillow 8.3.1 py38hee640a0_0 conda-forge pint 0.17 pypi_0 pypi pip 21.2.1 pypi_0 pypi platformdirs 2.2.0 pypi_0 pypi pluggy 0.13.1 py38h50d1736_4 conda-forge pooch 1.4.0 pyhd8ed1ab_0 conda-forge pre-commit 2.13.0 pypi_0 pypi psutil 5.8.0 py38h96a0964_1 conda-forge py 1.10.0 pyhd3deb0d_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pydantic 1.8.2 pypi_0 pypi pydeprecate 0.3.0 pypi_0 pypi pydicom 2.1.2 pyhd3deb0d_0 conda-forge pyopengl 3.1.5 pypi_0 pypi pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyqt5 5.15.4 pypi_0 pypi pyqt5-qt5 5.15.2 pypi_0 pypi pyqt5-sip 12.9.0 pypi_0 pypi pyro-api 0.1.2 pypi_0 pypi pyro-ppl 1.6.0 pypi_0 pypi pysocks 1.7.1 py38h50d1736_3 conda-forge pytest 6.2.4 py38h50d1736_0 conda-forge python 3.8.10 h3a889e0_1_cpython conda-forge python-bioformats 4.0.0 pypi_0 pypi python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-javabridge 4.0.0 pypi_0 pypi python-spams 2.6.1.11 pypi_0 pypi python_abi 3.8 2_cp38 conda-forge pytorch 1.8.0 cpu_py38h561cec5_1 conda-forge pytorch-lightning 1.3.8 pypi_0 pypi pytz 2021.1 pyhd8ed1ab_0 conda-forge pywavelets 1.1.1 py38hc7193ba_3 conda-forge pyyaml 5.4.1 py38h5406a74_0 conda-forge readline 8.1 h05e3726_0 conda-forge requests 2.26.0 pyhd8ed1ab_0 conda-forge rich 10.4.0 pypi_0 pypi rpy2 3.4.5 pypi_0 pypi s3transfer 0.4.2 pypi_0 pypi scanorama 1.7.1 pypi_0 pypi scikit-image 0.18.1 py38he9f00de_0 conda-forge scikit-learn 0.24.2 pypi_0 pypi scikit-misc 0.1.4 pypi_0 pypi scipy 1.5.4 pypi_0 pypi scvi-tools 0.11.0 pypi_0 pypi setuptools 49.6.0 py38h50d1736_3 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sleef 3.5.1 h35c211d_1 conda-forge snappy 1.1.8 hb1e8313_3 conda-forge snowballstemmer 2.1.0 pypi_0 pypi sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge spams 2.6.2.5 pypi_0 pypi sparse 0.12.0 pypi_0 pypi sphinx 4.0.2 pypi_0 pypi sphinx-rtd-theme 0.5.2 pypi_0 pypi sphinxcontrib-applehelp 1.0.2 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.0 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlite 3.36.0 h23a322b_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge tensorboard 2.4.1 pypi_0 pypi tensorflow 2.4.1 pypi_0 pypi tensorflow-estimator 2.4.0 pypi_0 pypi tensorly 0.6.0 pypi_0 pypi tifffile 2021.7.2 pyhd8ed1ab_0 conda-forge tk 8.6.10 h0419947_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge toolz 0.11.1 py_0 conda-forge torch 1.9.0 pypi_0 pypi torchmetrics 0.4.0 pypi_0 pypi tornado 6.1 py38h96a0964_1 conda-forge typing_extensions 3.10.0.0 pyha770c72_0 conda-forge urllib3 1.26.6 pyhd8ed1ab_0 conda-forge virtualenv 20.7.0 pypi_0 pypi vispy 0.7.1 pypi_0 pypi wheel 0.36.2 pyhd3deb0d_0 conda-forge xz 5.2.5 haf1e3a3_1 conda-forge yaml 0.2.5 haf1e3a3_0 conda-forge zfp 0.5.5 he49afe7_5 conda-forge zict 2.0.0 py_0 conda-forge zlib 1.2.11 h7795811_1010 conda-forge zstd 1.5.0 h582d3a0_0 conda-forge

ryanccarelli commented 3 years ago

I upgraded h5py to the latest version since it fixes this error and is compatible with other packages.