search

conda-forge / hdf5-feedstock

A conda-smithy repository for hdf5.
BSD 3-Clause "New" or "Revised" License
13 stars 57 forks source link

SZIP/libaec support on Windows #197

Closed mkitti closed 1 year ago

mkitti commented 1 year ago

Solution to issue cannot be found in the documentation.

Issue

Refiling of https://github.com/conda-forge/h5py-feedstock/issues/63 to here

This is a known issue from https://github.com/conda-forge/hdf5-feedstock/pull/179#issuecomment-1207396346 and the residual of #120 .

I'm noting that MSYS2 has a patch that possibly addresses this issue. https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-hdf5/hdf5-fix-find-szip.patch https://packages.msys2.org/base/mingw-w64-hdf5

Upstream appears to have fixed this issue. https://github.com/HDFGroup/hdf5/pull/2492

Installed packages

bzip2              conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
  ca-certificates    conda-forge/win-64::ca-certificates-2022.12.7-h5b45459_0
  cached-property    conda-forge/noarch::cached-property-1.5.2-hd8ed1ab_1
  cached_property    conda-forge/noarch::cached_property-1.5.2-pyha770c72_1
  h5py               conda-forge/win-64::h5py-3.8.0-nompi_py311hfbc32a6_101
  hdf5               conda-forge/win-64::hdf5-1.14.0-nompi_h5af7005_102
  intel-openmp       conda-forge/win-64::intel-openmp-2023.0.0-h57928b3_25922
  krb5               conda-forge/win-64::krb5-1.20.1-heb0366b_0
  libblas            conda-forge/win-64::libblas-3.9.0-16_win64_mkl
  libcblas           conda-forge/win-64::libcblas-3.9.0-16_win64_mkl
  libcurl            conda-forge/win-64::libcurl-7.88.1-h68f0423_0
  libffi             conda-forge/win-64::libffi-3.4.2-h8ffe710_5
  libhwloc           conda-forge/win-64::libhwloc-2.9.0-h51c2c0f_0
  libiconv           conda-forge/win-64::libiconv-1.17-h8ffe710_0
  liblapack          conda-forge/win-64::liblapack-3.9.0-16_win64_mkl
  libsqlite          conda-forge/win-64::libsqlite-3.40.0-hcfcfb64_0
  libssh2            conda-forge/win-64::libssh2-1.10.0-h9a1e1f7_3
  libxml2            conda-forge/win-64::libxml2-2.10.3-hc3477c8_0
  libzlib            conda-forge/win-64::libzlib-1.2.13-hcfcfb64_4
  mkl                conda-forge/win-64::mkl-2022.1.0-h6a75c08_874
  numpy              conda-forge/win-64::numpy-1.24.2-py311h0b4df5a_0
  openssl            conda-forge/win-64::openssl-3.0.8-hcfcfb64_0
  pip                conda-forge/noarch::pip-23.0.1-pyhd8ed1ab_0
  pthreads-win32     conda-forge/win-64::pthreads-win32-2.9.1-hfa6e2cd_3
  python             conda-forge/win-64::python-3.11.0-hcf16a7b_0_cpython
  python_abi         conda-forge/win-64::python_abi-3.11-3_cp311
  setuptools         conda-forge/noarch::setuptools-67.5.1-pyhd8ed1ab_0
  tbb                conda-forge/win-64::tbb-2021.8.0-h91493d7_0
  tk                 conda-forge/win-64::tk-8.6.12-h8ffe710_0
  tzdata             conda-forge/noarch::tzdata-2022g-h191b570_0
  ucrt               conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_0
  vc                 conda-forge/win-64::vc-14.3-hb6edc58_10
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.34.31931-h4c5c07a_10
  wheel              conda-forge/noarch::wheel-0.38.4-pyhd8ed1ab_0
  xz                 conda-forge/win-64::xz-5.2.6-h8d14728_0

Environment info

(h5py_hdf5_test) C:\>conda info

     active environment : h5py_hdf5_test
    active env location : C:\Users\_\.julia\conda\3\envs\h5py_hdf5_test
            shell level : 2
       user config file : C:\Users\_\.condarc
 populated config files :
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.9.5.final.0
       virtual packages : __cuda=11.7=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\_\.julia\conda\3  (writable)
      conda av data dir : C:\Users\_\.julia\conda\3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\_\.julia\conda\3\pkgs
                          C:\Users\_\.conda\pkgs
                          C:\Users\_\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\_\.julia\conda\3\envs
                          C:\Users\_\.conda\envs
                          C:\Users\_\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.10.3 requests/2.25.1 CPython/3.9.5 Windows/10 Windows/10.0.19044
          administrator : False
             netrc file : None
           offline mode : False
mkitti commented 1 year ago

The observation that the libaec builds on Windows appear to be working in https://github.com/zarr-developers/numcodecs/pull/422 suggests the problem is mostly with the cmake build system.

A small patch similar to the MSYS2 patch may be successful. Alternatively, we wait for upstream which should have a patch release near the beginning of May 2023 according to the HDF Group release schedule.

jakirkham commented 1 year ago

Would suggest trying to drop the comment and selector here and bump the build/number

https://github.com/conda-forge/hdf5-feedstock/blob/085d05267f37424fc40032ea5bd77ca7384eb05d/recipe/meta.yaml#L85-L86

Should add currently there are warnings like this on Windows:

CMake Warning:
-- Build files have been written to: D:/bld/hdf5_1675776169379/work/build
  Manually-specified variables were not used by the project:

    SZIP_INCLUDE_DIR
    SZIP_LIBRARY

So lines like these probably need to be reworked or dropped:

https://github.com/conda-forge/hdf5-feedstock/blob/085d05267f37424fc40032ea5bd77ca7384eb05d/recipe/bld.bat#L23-L24

mkitti commented 1 year ago

Steps to reproduce the problem.

>>> import h5py

>>> with h5py.File("test.h5", "w") as h5f:
...     ds = h5f.create_dataset("test", (1024, 1024), chunks=(16,16), dtype="uint16", compression="szip", compression_opts=('ec',32))
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "C:\Users\_\.julia\conda\3\envs\h5py_hdf5_test\Lib\site-packages\h5py\_hl\group.py", line 183, in create_dataset
    dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\_\.julia\conda\3\envs\h5py_hdf5_test\Lib\site-packages\h5py\_hl\dataset.py", line 106, in make_new_dset
    dcpl = filters.fill_dcpl(
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\_\.julia\conda\3\envs\h5py_hdf5_test\Lib\site-packages\h5py\_hl\filters.py", line 185, in fill_dcpl
    raise ValueError('Compression filter "%s" is unavailable' % compression)
ValueError: Compression filter "szip" is unavailable

>>> h5py.h5z.filter_avail(4)
False