search

nomad-coe / nomad

NOMAD lets you manage and share your materials science data in a way that makes it truly useful to you, your group, and the community.
https://nomad-lab.eu
Apache License 2.0
64 stars 14 forks source link

Install problems #98

Closed helgestein closed 3 months ago

helgestein commented 4 months ago

On macOS you cannot install nomad-lab>=1.2.0 because of the pymatgen depency pymatgen==2023.2.28 failing to build a wheel causing the entire install to fail. The same happens in any docker container run on the apple silicon platform.

Simple way of reproducing this error on a M1+ chip is to make a minimal python3 docker file with RUN pip install nomad-lab

lauri-codes commented 4 months ago

Hi @helgestein!

Could you share the install logs from your attempt to install nomad-lab=1.2.0 on Apple silicon? Have you succeeded in installing an older version of nomad-lab or pymatgen on Apple silicon?

It would also be interesting to see what kind of errors you get with docker when trying to run our docker image (the latests one is: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:latest): in principle, the issue in docker should be something else, since the wheel is already built for you.

I hope we can find some solution, but it is certainly a challenge to ensure support for Apple silicon, since we can't e.g. run systematic installation tests on this specialized hardware.

helgestein commented 4 months ago

Sure! We should maybe split the docker and pip discussion simply because running the docker on most laptops will eat too much resources. See also the general comment on dependencies below.

so running pip install nomad-lab on macOS Sonoma 14.1.1 on a M2 chipset on python 3.10.10 results in: (truncating the output)
Error compiling Cython file:

  ...
      #ensure correct shape
      fcoords1, fcoords2 = np.atleast_2d(fcoords1, fcoords2)

      pbc = lattice.pbc
      cdef int n_pbc = sum(pbc)
      cdef int n_pbc_im = 3 ** n_pbc
                            ^
  ------------------------------------------------------------

  pymatgen/util/coord_cython.pyx:93:26: Cannot assign type 'double' to 'int'
  Compiling pymatgen/util/coord_cython.pyx because it changed.
  [1/1] Cythonizing pymatgen/util/coord_cython.pyx
  Traceback (most recent call last):
    File "/Users/helgesorenstein/mambaforge/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/Users/helgesorenstein/mambaforge/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/helgesorenstein/mambaforge/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 410, in build_wheel
      return self._build_with_temp_dir(
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 26, in <module>
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
      return distutils.core.setup(**attrs)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/normal/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 368, in run
      self.run_command("build")
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 963, in run_command
      super().run_command(command)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 89, in run
      _build_ext.run(self)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 250, in build_extension
      _build_ext.build_extension(self, ext)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/Cython/Distutils/build_ext.py", line 130, in build_extension
      new_ext = cythonize(
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
      cythonize_one(*args)
    File "/private/var/folders/gf/ksqjl6fj4zj55hdfzlttfvwr0000gn/T/pip-build-env-uxlrgg53/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
      raise CompileError(None, pyx_file)
  Cython.Compiler.Errors.CompileError: pymatgen/util/coord_cython.pyx
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pymatgen Failed to build pymatgen ERROR: Could not build wheels for pymatgen, which is required to install pyproject.toml-based projects

running pip install nomad-lab -prefer-binary results in the same issue. Installing the latest pymatgen version (2023.5.10) via pip install pymatgen works just fine. It really seems like that exact version of pymatgen that you have as a dependency (2023.2.28) fails.

Overall, I kind of disagree with the necessity to install all kinds of theory-focused dependencies when the goal is to write and test a parser (especially as an experimentalist) for a nomad-oasis. I'm guessing your goal is widespread adoption beyond computational chemistry. Another issue here is that phonopy is also causing issues upon installation on windows machines as it nessecitates a VS C++ compiler. These errors upon installing make the entry barrier even higher for new adopters outside computational chemistry.

helgestein commented 4 months ago

Ok so creating a conda venv with python 3.9 will at least install nomad-lab, a nomad import will work BUT trying to start to develop an EntryArchive will fail because the numpy version installed as part of nomad-lab will result in:

Python 3.9.0 | packaged by conda-forge | (default, Nov 26 2020, 07:55:15) [Clang 11.0.0 ] on darwin Type "help", "copyright", "credits" or "license" for more information.

import numpy as np np.version '1.22.4' from nomad.datamodel import EntryArchive, EntryMetadata

...

raise AttributeError("module {!r} has no attribute "

AttributeError: module 'numpy' has no attribute 'float128'

numpy in that version does have numpy.longdouble, which is the same as float128, but float128 is not available on macros/numpy. When installing in Python 3.9, updating numpy will not work besides numpy's latest version on macos and python 3.11.8 does not have float128 either. Besides I cannot really think of a utter nessesity to carry hat level of precision in any matsci experiment.

The code in the utils functions init does the following: float_numpy = {np.float16, np.float32, np.float64} | (set() if os.name == 'nt' else {np.float128} so maybe add a try except with np.longdouble?

lauri-codes commented 4 months ago

Thanks for the additional details and comments!

  1. Installing the latest pymatgen version (2023.5.10) via pip install pymatgen works just fine. It really seems like that exact version of pymatgen that you have as a dependency (2023.2.28) fails.

    This might have something to do with Cython going through some breaking changes when migrating to 0.30, which has caused us some trouble previously as well. I think we can quite easily bump up the version number to see if this helps.

  2. Overall, I kind of disagree with the necessity to install all kinds of theory-focused dependencies when the goal is to write and test a parser (especially as an experimentalist) for a nomad-oasis. I'm guessing your goal is widespread adoption beyond computational chemistry.

    You are definitely right here. Historically NOMAD has been very theory-focused, thus the current dependencies. But we are actively migrating to a more plugin-based approach, where the idea is to keep the main package as lean as possible, and then users can bring in additional dependencies through plugins. Unfortunately, this is not yet done, but is underway. With these changes, also phonopy will become a part of an optional plugin. I can check if there is an intermediate solution where phonopy gets installed only if the user requires the parsing extras with pip install nomad[infractructure, parsing].

  3. When installing in Python 3.9, updating numpy will not work besides numpy's latest version on macos and python 3.11.8 does not have float128 either. Besides I cannot really think of a utter nessesity to carry hat level of precision in any matsci experiment.

    I agree that float128 should no be required by default: I don't think I have seen anyone use it, but it was included just in case. We can make it optionally with a try/catch block as you suggest.

Overall, I will see if I can address 1,3 and the phonopy problem in a new MR. If you have then time, you could test these changes to see if your problems get resolved or if additional steps are needed.

lauri-codes commented 4 months ago

I have now merged a proposed fix into the develop branch.

Unfortunately, I can't test it thoroughly since I don't have the hardware available. If you have time, you could try to clone the develop branch and install the nomad-lab package from there with pip install . This is not exactly the same as installing our PyPI package, but would at least indicate if some of the problems are gone. There is also an updated docker image at: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:develop.

lauri-codes commented 3 months ago

Closing for now.