Failed to build ubermag in the new environment

code-whale commented 2 years ago

Dear Ubermag Team， I'm trying to build a new conda environment to use the latest version of ubermag, but the test failed after installation. Do you know why? I use the Mac OS 12 system, and the installation code is:

conda create -n ubermag2 python=3.8
conda install --channel conda-forge ubermag

Errors will be reported when running tests conda install --channel conda-forge ubermag and typing something likeimport oommfc as oc in jupyter. The test return error is as follows:

(ubermag2) whalez@MacBook-Pro ~ % python -c "import ubermag; ubermag.test()"
============================= test session starts ==============================
platform darwin -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /Users/whalez/miniconda3/envs/ubermag2/bin/python
cachedir: .pytest_cache
rootdir: /Users/whalez
plugins: anyio-3.6.1
collected 0 items / 1 error                                                    

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
miniconda3/envs/ubermag2/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
        level      = 0
        name       = 'oommfc.tests.conftest'
        package    = None
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
        level      = 0
        name       = 'oommfc.tests.conftest'
        package    = None
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
        import_    = <function _gcd_import at 0x7f8a758884c0>
        module     = <object object at 0x7f8a7585f060>
        name       = 'oommfc.tests.conftest'
<frozen importlib._bootstrap>:961: in _find_and_load_unlocked
    ???
        import_    = <function _gcd_import at 0x7f8a758884c0>
        name       = 'oommfc.tests.conftest'
        parent     = 'oommfc.tests'
        path       = None
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
        args       = ('oommfc.tests',)
        f          = <function _gcd_import at 0x7f8a758884c0>
        kwds       = {}
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
        level      = 0
        name       = 'oommfc.tests'
        package    = None
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
        import_    = <function _gcd_import at 0x7f8a758884c0>
        module     = <object object at 0x7f8a7585f060>
        name       = 'oommfc.tests'
<frozen importlib._bootstrap>:961: in _find_and_load_unlocked
    ???
        import_    = <function _gcd_import at 0x7f8a758884c0>
        name       = 'oommfc.tests'
        parent     = 'oommfc'
        path       = None
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
        args       = ('oommfc',)
        f          = <function _gcd_import at 0x7f8a758884c0>
        kwds       = {}
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
        level      = 0
        name       = 'oommfc'
        package    = None
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
        import_    = <function _gcd_import at 0x7f8a758884c0>
        module     = <object object at 0x7f8a7585f060>
        name       = 'oommfc'
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
        import_    = <function _gcd_import at 0x7f8a758884c0>
        name       = 'oommfc'
        parent     = ''
        path       = None
        spec       = ModuleSpec(name='oommfc', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362280>, origin='/User...__init__.py', submodule_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc'])
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
        module     = <module 'oommfc' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/__init__.py'>
        spec       = ModuleSpec(name='oommfc', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362280>, origin='/User...__init__.py', submodule_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc'])
<frozen importlib._bootstrap_external>:843: in exec_module
    ???
        code       = <code object <module> at 0x7f8a7a32ea80, file "/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/__init__.py", line 1>
        module     = <module 'oommfc' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/__init__.py'>
        self       = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362280>
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
        args       = (<code object <module> at 0x7f8a7a32ea80, file "/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oom...alculator.', '__file__': '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/__init__.py', ...})
        f          = <built-in function exec>
        kwds       = {}
miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/__init__.py:5: in <module>
    import oommfc.oommf
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/__pycache__/__init__.cpython-38.pyc'
        __doc__    = 'OOMMF calculator.'
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/__init__.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362280>
        __name__   = 'oommfc'
        __package__ = 'oommfc'
        __path__   = ['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc']
        __spec__   = ModuleSpec(name='oommfc', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362280>, origin='/User...__init__.py', submodule_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc'])
        pkg_resources = <module 'pkg_resources' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/pkg_resources/__init__.py'>
        pytest     = <module 'pytest' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/pytest/__init__.py'>
miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf/__init__.py:2: in <module>
    from .oommf import (
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf/__pycache__/__init__.cpython-38.pyc'
        __doc__    = 'OOMMF driving utility'
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf/__init__.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362550>
        __name__   = 'oommfc.oommf'
        __package__ = 'oommfc.oommf'
        __path__   = ['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf']
        __spec__   = ModuleSpec(name='oommfc.oommf', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362550>, origin=...__.py', submodule_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf'])
miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf/oommf.py:10: in <module>
    import micromagneticmodel as mm
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf/__pycache__/oommf.cpython-38.pyc'
        __doc__    = None
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf/oommf.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362790>
        __name__   = 'oommfc.oommf.oommf'
        __package__ = 'oommfc.oommf'
        __spec__   = ModuleSpec(name='oommfc.oommf.oommf', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a362790>, origin='/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/oommfc/oommf/oommf.py')
        abc        = <module 'abc' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/abc.py'>
        datetime   = <module 'datetime' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/datetime.py'>
        logging    = <module 'logging' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/logging/__init__.py'>
        os         = <module 'os' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/os.py'>
        shutil     = <module 'shutil' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/shutil.py'>
        sp         = <module 'subprocess' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/subprocess.py'>
        sys        = <module 'sys' (built-in)>
        time       = <module 'time' (built-in)>
miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/__init__.py:7: in <module>
    import micromagneticmodel.examples
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/__pycache__/__init__.cpython-38.pyc'
        __doc__    = 'Domain-specific language for computational magnetism.'
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/__init__.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a367310>
        __name__   = 'micromagneticmodel'
        __package__ = 'micromagneticmodel'
        __path__   = ['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel']
        __spec__   = ModuleSpec(name='micromagneticmodel', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7a367310>, o..., submodule_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel'])
        abstract   = <module 'micromagneticmodel.abstract' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/abstract/__init__.py'>
        consts     = <module 'micromagneticmodel.consts' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/consts.py'>
        micromagneticmodel = <module 'micromagneticmodel' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/__init__.py'>
        pkg_resources = <module 'pkg_resources' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/pkg_resources/__init__.py'>
        pytest     = <module 'pytest' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/pytest/__init__.py'>
miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples/__init__.py:2: in <module>
    from .examples import macrospin
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples/__pycache__/__init__.cpython-38.pyc'
        __doc__    = 'Functions for creating examples'
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples/__init__.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7b9c2eb0>
        __name__   = 'micromagneticmodel.examples'
        __package__ = 'micromagneticmodel.examples'
        __path__   = ['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples']
        __spec__   = ModuleSpec(name='micromagneticmodel.examples', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7b9...le_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples'])
miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples/examples.py:1: in <module>
    import discretisedfield as df
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples/__pycache__/examples.cpython-38.pyc'
        __doc__    = None
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples/examples.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7b9dc160>
        __name__   = 'micromagneticmodel.examples.examples'
        __package__ = 'micromagneticmodel.examples'
        __spec__   = ModuleSpec(name='micromagneticmodel.examples.examples', loader=<_frozen_importlib_external.SourceFileLoader object at ...>, origin='/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/micromagneticmodel/examples/examples.py')
miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield/__init__.py:8: in <module>
    from .field import Field
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield/__pycache__/__init__.cpython-38.pyc'
        __doc__    = 'Finite-difference fields.'
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield/__init__.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7b9dc400>
        __name__   = 'discretisedfield'
        __package__ = 'discretisedfield'
        __path__   = ['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield']
        __spec__   = ModuleSpec(name='discretisedfield', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7b9dc400>, ori...y', submodule_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield'])
        os         = <module 'os' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/os.py'>
        pkg_resources = <module 'pkg_resources' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/pkg_resources/__init__.py'>
        plt        = <module 'matplotlib.pyplot' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/matplotlib/pyplot.py'>
        pytest     = <module 'pytest' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/pytest/__init__.py'>
miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield/field.py:9: in <module>
    import h5py
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield/__pycache__/field.cpython-38.pyc'
        __doc__    = None
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield/field.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7cdb5490>
        __name__   = 'discretisedfield.field'
        __package__ = 'discretisedfield'
        __spec__   = ModuleSpec(name='discretisedfield.field', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7cdb5490>, origin='/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/discretisedfield/field.py')
        collections = <module 'collections' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/collections/__init__.py'>
        functools  = <module 'functools' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/functools.py'>
        math       = <module 'math' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/lib-dynload/math.cpython-38-darwin.so'>
        numbers    = <module 'numbers' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/numbers.py'>
        re         = <module 're' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/re.py'>
        struct     = <module 'struct' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/struct.py'>
        warnings   = <module 'warnings' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/warnings.py'>
miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/__init__.py:33: in <module>
    from . import version
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/__pycache__/__init__.cpython-38.pyc'
        __doc__    = '\n    This is the h5py package, a Python interface to the HDF5\n    scientific data format.\n'
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/__init__.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7e607940>
        __name__   = 'h5py'
        __package__ = 'h5py'
        __path__   = ['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py']
        __spec__   = ModuleSpec(name='h5py', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7e607940>, origin='/Users/...y/__init__.py', submodule_search_locations=['/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py'])
        _errors    = <module 'h5py._errors' from '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/_errors.cpython-38-darwin.so'>
        _warn      = <built-in function warn>
        atexit     = <module 'atexit' (built-in)>
miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/version.py:15: in <module>
    from . import h5 as _h5
        __builtins__ = <builtins>
        __cached__ = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/__pycache__/version.cpython-38.pyc'
        __doc__    = '\n    Versioning module for h5py.\n'
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/version.py'
        __loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f8a7e6112b0>
        __name__   = 'h5py.version'
        __package__ = 'h5py'
        __spec__   = ModuleSpec(name='h5py.version', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f8a7e6112b0>, origin='/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/version.py')
        namedtuple = <function namedtuple at 0x7f8a75ce4f70>
h5py/h5.pyx:1: in init h5py.h5
    ???
E   ImportError: dlopen(/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/defs.cpython-38-darwin.so, 0x0002): Symbol not found: _H5Pget_fapl_ros3
E     Referenced from: /Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/defs.cpython-38-darwin.so
E     Expected in: /Users/whalez/miniconda3/envs/ubermag2/lib/libhdf5.103.dylib
        H5PYConfig = <class 'h5py.h5.H5PYConfig'>
        __builtins__ = <builtins>
        __doc__    = None
        __file__   = '/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/h5.cpython-38-darwin.so'
        __loader__ = <_frozen_importlib_external.ExtensionFileLoader object at 0x7f8a7e611520>
        __name__   = 'h5py.h5'
        __package__ = 'h5py'
        __pyx_capi__ = {'get_config': <capsule object "struct __pyx_obj_4h5py_2h5_H5PYConfig *(int __pyx_skip_dispatch)" at 0x7f8a7e611810>}
        __spec__   = ModuleSpec(name='h5py.h5', loader=<_frozen_importlib_external.ExtensionFileLoader object at 0x7f8a7e611520>, origin='/Users/whalez/miniconda3/envs/ubermag2/lib/python3.8/site-packages/h5py/h5.cpython-38-darwin.so')
        get_config = <built-in function get_config>
=========================== short test summary info ============================
ERROR  - ImportError: dlopen(/Users/whalez/miniconda3/envs/ubermag2/lib/pytho...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.37s ===============================

Can you tell me how to solve it? Is there something wrong with the code I built? Looking forward to your reply！ Best regards code-whale

lang-m commented 2 years ago

Hi @code-whale, that looks like a problem with h5py and conda (there is an open issue in conda-forge/h5py-feedstock#103).

I could think of two different possible workarounds:

Explicitly install h5py in the version that you have in your working environment (conda list shows all installed packages)
or try to install h5py via pip in your current environment. Apparently the PyPI version cannot have this problem.

@marijanbeg Can you reproduce this?

code-whale commented 2 years ago

Hi @lang-m ,Thank you for your reply. I'll try to solve this problem again. Here's what I've tried: A Install pip install h5py in the ubermag2 environment according to your instructions, as shown below：

(ubermag2) whalez@MacBook-Pro envs % pip install h5py
Requirement already satisfied: h5py in ./ubermag2/lib/python3.8/site-packages (3.2.1)
Requirement already satisfied: numpy>=1.16.0 in ./ubermag2/lib/python3.8/site-packages (from h5py) (1.22.3)

I installed ubermag in Windows11 and did not show any problems I performed a conda update on all packages in the original ubermag environment - all showed no problems and the upgrade was successful. Ubermag is now version 0.62.1. I modified the conda source priority and replaced it all with conda-forge. The installation test is being carried out to detect if the problem occurs with the Conda source priority or Mac OS.

lang-m commented 2 years ago

It is quite possible that the current problem only affects Mac.

My pip suggestion was missing a flag to force re-installation (pip install h5py --force-reinstall). Otherwise, it is not actually doing anything. Problem: I am not sure if this will also re-install numpy and then break at a different point.

code-whale commented 2 years ago

@lang-m , I seem to find the problem. As you said, the problem appears on h5py. Some of my conda sources are not conda-forge, but mirrored (in order to improve the download speed), so I installed the h5py version is 3.2.1, while the h5py version of ubermag in conda-forge is 3.6, so there is a mismatch. I chose the following two methods: In the case of a non-conda-forge source installation, using pip install-upgrade h5py in the new environment after installation can solve the problem after testing.

The other is to modify the source (which will affect the download speed in some areas): conda config--add channels conda-forge and then conda config --set channel_priority strict .This method comes from the suggestion of @ZhiyuZhang001. Both of the above methods have successfully solved the problem, and you can turn it off at any time. Thank you again for your help!

lang-m commented 2 years ago

TLDR for people coming here in the future: h5py via conda does not strictly enforce a compatible hdf5 library during installation. To avoid problems make sure that h5py and hdf5 are installed from the same channel and compatible (or use pip for installing h5py as it brings a compatible hdf5 library).

ubermag / help

Failed to build ubermag in the new environment #206