Error with `pip install nomad-lab[parsing]` due to `netCDF4==1.5.4`

[Andrew-S-Rosen]

I tried running pip install nomad-lab[parsing] and got the following error. Running pip install nomad-lab works just fine.

Apparently, pip install netCDF4==1.5.4 is the culprit. This is in a fresh Python 3.10 Conda environment on Ubuntu. Using a newer version of netCDF4 resolves the issue.

In connection with https://github.com/openjournals/joss-reviews/issues/5388#issuecomment-1548824444, I thought the version pinning was reduced overall. Was there a specific need for 1.5.4 here?

Collecting nomad-lab[parsing]
  Using cached nomad-lab-1.2.0.tar.gz (21.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy~=1.22.4 (from nomad-lab[parsing])
  Using cached numpy-1.22.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Collecting nptyping~=1.4.4 (from nomad-lab[parsing])
  Using cached nptyping-1.4.4-py3-none-any.whl (31 kB)
Collecting pandas<2.0.0,>=1.3.5 (from nomad-lab[parsing])
  Using cached pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB)
Collecting cachetools>=4.2.4 (from nomad-lab[parsing])
  Using cached cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Collecting docstring-parser>=0.12 (from nomad-lab[parsing])
  Using cached docstring_parser-0.15-py3-none-any.whl (36 kB)
Collecting pint==0.17 (from nomad-lab[parsing])
  Using cached Pint-0.17-py2.py3-none-any.whl (204 kB)
Collecting orjson==3.6.0 (from nomad-lab[parsing])
  Using cached orjson-3.6.0-cp310-cp310-manylinux_2_24_x86_64.whl (235 kB)
Collecting click>=7.1.2 (from nomad-lab[parsing])
  Using cached click-8.1.4-py3-none-any.whl (98 kB)
Collecting requests>=2.27.1 (from nomad-lab[parsing])
  Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Collecting pytz>=2022.7.1 (from nomad-lab[parsing])
  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting aniso8601>=7.0.0 (from nomad-lab[parsing])
  Using cached aniso8601-9.0.1-py2.py3-none-any.whl (52 kB)
Collecting ase~=3.19.0 (from nomad-lab[parsing])
  Using cached ase-3.19.3-py3-none-any.whl (2.1 MB)
Collecting python-keycloak>=0.26.1 (from nomad-lab[parsing])
  Using cached python_keycloak-3.3.0-py3-none-any.whl (58 kB)
Collecting elasticsearch-dsl==7.4.0 (from nomad-lab[parsing])
  Using cached elasticsearch_dsl-7.4.0-py2.py3-none-any.whl (63 kB)
Collecting pydantic<2.0.0,>=1.10.8 (from nomad-lab[parsing])
  Using cached pydantic-1.10.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting jmespath>=0.10.0 (from nomad-lab[parsing])
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting httpx>=0.23.3 (from nomad-lab[parsing])
  Using cached httpx-0.24.1-py3-none-any.whl (75 kB)
Collecting memoization>=0.4.0 (from nomad-lab[parsing])
  Using cached memoization-0.4.0.tar.gz (41 kB)
  Preparing metadata (setup.py) ... done
Collecting rfc3161ng>=2.1.3 (from nomad-lab[parsing])
  Using cached rfc3161ng-2.1.3-py2.py3-none-any.whl (9.9 kB)
Collecting lxml>=4.6 (from nomad-lab[parsing])
  Using cached lxml-4.9.3-cp310-cp310-manylinux_2_28_x86_64.whl (7.9 MB)
Collecting wrapt>=1.12.1 (from nomad-lab[parsing])
  Using cached wrapt-1.15.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
Collecting pyyaml>=6.0 (from nomad-lab[parsing])
  Using cached PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
Collecting python-magic==0.4.24 (from nomad-lab[parsing])
  Using cached python_magic-0.4.24-py2.py3-none-any.whl (12 kB)
Collecting hjson>=3.0.2 (from nomad-lab[parsing])
  Using cached hjson-3.1.0-py3-none-any.whl (54 kB)
Collecting scipy>=1.7.1 (from nomad-lab[parsing])
  Using cached scipy-1.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
Collecting panedr>=0.2 (from nomad-lab[parsing])
  Using cached panedr-0.7.1-py3-none-any.whl (15 kB)
Collecting parmed>=3.0.0 (from nomad-lab[parsing])
  Using cached ParmEd-4.1.0.tar.gz (2.2 MB)
  Preparing metadata (setup.py) ... done
Collecting mdanalysis==2.5.0 (from nomad-lab[parsing])
  Using cached MDAnalysis-2.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.0 MB)
Collecting xarray>=0.20.2 (from nomad-lab[parsing])
  Using cached xarray-2023.6.0-py3-none-any.whl (999 kB)
Collecting phonopy~=2.11.0 (from nomad-lab[parsing])
  Using cached phonopy-2.11.0.tar.gz (3.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pymatgen~=2023.2.28 (from nomad-lab[parsing])
  Using cached pymatgen-2023.2.28-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.4 MB)
Collecting bitarray>=2.3.5 (from nomad-lab[parsing])
  Using cached bitarray-2.7.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (273 kB)
Collecting scikit-learn>=1.0.2 (from nomad-lab[parsing])
  Using cached scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.8 MB)
Collecting toposort (from nomad-lab[parsing])
  Using cached toposort-1.10-py3-none-any.whl (8.5 kB)
Collecting openpyxl>=3.0.0 (from nomad-lab[parsing])
  Using cached openpyxl-3.1.2-py2.py3-none-any.whl (249 kB)
Collecting asr==0.4.1 (from nomad-lab[parsing])
  Using cached asr-0.4.1-py3-none-any.whl (274 kB)
Collecting netCDF4==1.5.4 (from nomad-lab[parsing])
  Using cached netCDF4-1.5.4.tar.gz (793 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-a_n4w42k/netcdf4_5458dabc85c34e0288c27190173c88bb/setup.py", line 397, in <module>
          _populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs)
        File "/tmp/pip-install-a_n4w42k/netcdf4_5458dabc85c34e0288c27190173c88bb/setup.py", line 345, in _populate_hdf5_info
          raise ValueError('did not find HDF5 headers')
      ValueError: did not find HDF5 headers
      reading from setup.cfg...

          HDF5_DIR environment variable not set, checking some standard locations ..
      checking /home/rosen ...
      checking /usr/local ...
      checking /sw ...
      checking /opt ...
      checking /opt/local ...
      checking /usr ...
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[markus1978]

Please not that you most likely don't need the [parsing] extra. Most parsers will work just fine without. The parsing extras are only required for hfd5 and netCDF based codes and some optional normalising functionality like the new SOAP descriptor.

Unfortunately, there is a limit to what we can do here. The version restrictions we removed, are already a big trade-off between easy installs and the risks of new dependency versions introducing breaking changes. For example you a couple a days ago, the pydantic 2.0.0 release was breaking our build because we removed version restriction. We removed most version restrictions on the normal install, but i am afraid we need to keep them for the extra dependencies that are most likely only affecting the service operations anyways.

[Andrew-S-Rosen]

@markus1978 --- thanks, that makes sense! Anyway, the more important was more about just letting you all know that the pip install isn't working as expected, but that will presumably be resolved once the netCDF version is changed.

[markus1978]

Of course, i just wanted to explain why we have not just removed every version restriction. We will test and see what happens if we pump the netCDF version.