search

dkriegner / xrayutilities

xrayutilities - a package with useful scripts for X-ray diffraction physicists
http://xrayutilities.sourceforge.io
GNU General Public License v2.0
81 stars 29 forks source link

Import error with numpy.core.multiarray #174

Closed RubelMozumder closed 4 months ago

RubelMozumder commented 9 months ago

Getting import error as follows,

ImportError while importing test module 
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/dataconverter/test_readers.py:66: in <module>
    @pytest.mark.parametrize("reader", get_all_readers())
tests/dataconverter/test_readers.py:55: in get_all_readers
    for reader in [get_reader(x) for x in get_names_of_all_readers()]:
tests/dataconverter/test_readers.py:55: in <listcomp>
    for reader in [get_reader(x) for x in get_names_of_all_readers()]:
pynxtools/dataconverter/convert.py:50: in get_reader
    spec.loader.exec_module(module)  # type: ignore[attr-defined]
pynxtools/dataconverter/readers/xrd/reader.py:29: in <module>
    from pynxtools.dataconverter.readers.xrd.xrd_parser import parse_and_fill_template
pynxtools/dataconverter/readers/xrd/xrd_parser.py:24: in <module>
    from xrayutilities.io.panalytical_xml import XRDMLFile  # for reading XRDML files
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xrayutilities/__init__.py:32: in <module>
    from . import __path__, analysis, config, io, materials, math, simpack
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xrayutilities/analysis/__init__.py:27: in <module>
    from .line_cuts import (get_arbitrary_line, get_omega_scan, get_qx_scan,
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xrayutilities/analysis/line_cuts.py:[21](https://github.com/FAIRmat-NFDI/pynxtools/actions/runs/6983759730/job/19005448819?pr=179#step:7:22): in <module>
    from ..experiment import HXRD
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xrayutilities/experiment.py:[40](https://github.com/FAIRmat-NFDI/pynxtools/actions/runs/6983759730/job/19005448819?pr=179#step:7:41): in <module>
    from . import config, cxrayutilities, math, utilities
E   ImportError: numpy.core.multiarray failed to import

The pyhton env has been given bellow: python 3.8 and python packages

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --extra=dev --output-file=dev-requirements.txt pyproject.toml
#
appdirs==1.4.4
    # via requests-cache
asciitree==0.3.3
    # via zarr
ase==3.19.0
    # via
    #   ifes-apt-tc-data-modeling
    #   pynxtools (pyproject.toml)
asteval==0.9.29
    # via lmfit
astroid==2.15.5
    # via pylint
attrs==22.1.0
    # via
    #   cattrs
    #   requests-cache
backcall==0.2.0
    # via ipython
build==0.10.0
    # via pip-tools
cattrs==22.2.0
    # via requests-cache
certifi==2022.9.24
    # via requests
charset-normalizer==2.1.1
    # via requests
click==8.1.3
    # via
    #   pip-tools
    #   pynxtools (pyproject.toml)
cloudpickle==2.2.0
    # via dask
coverage[toml]==6.5.0
    # via pytest-cov
cycler==0.11.0
    # via matplotlib
dask[array]==2022.2.0
    # via
    #   hyperspy
    #   kikuchipy
    #   orix
    #   pyxem
debugpy==1.6.3
    # via ipykernel
decorator==5.1.1
    # via
    #   ipyparallel
    #   ipython
diffpy-structure==3.1.0
    # via
    #   diffsims
    #   kikuchipy
    #   orix
diffsims==0.5.1
    # via
    #   kikuchipy
    #   pyxem
dill==0.3.5.1
    # via
    #   hyperspy
    #   pylint
entrypoints==0.4
    # via
    #   ipyparallel
    #   jupyter-client
    #   numcodecs
exceptiongroup==1.1.0
    # via
    #   cattrs
    #   pytest
fabio==2023.4.1
    # via
    #   pyfai
    #   silx
fasteners==0.18
    # via zarr
flatdict==4.0.1
    # via pynxtools (pyproject.toml)
fonttools==4.37.4
    # via matplotlib
fsspec==2022.8.2
    # via
    #   dask
    #   hyperspy
future==0.18.3
    # via uncertainties
gitdb==4.0.10
    # via gitpython
gitpython==3.1.30
    # via pynxtools (pyproject.toml)
h5py==3.7.0
    # via
    #   hyperspy
    #   ifes-apt-tc-data-modeling
    #   kikuchipy
    #   nionswift
    #   orix
    #   pyfai
    #   pynxtools (pyproject.toml)
    #   silx
hyperspy==1.7.4
    # via
    #   kikuchipy
    #   pynxtools (pyproject.toml)
    #   pyxem
idna==3.4
    # via requests
ifes-apt-tc-data-modeling==0.0.9
    # via pynxtools (pyproject.toml)
imageio==2.22.1
    # via
    #   hyperspy
    #   kikuchipy
    #   nionswift
    #   nionswift-io
    #   nionui
    #   scikit-image
importlib-metadata==5.0.0
    # via hyperspy
iniconfig==2.0.0
    # via pytest
ipykernel==6.16.0
    # via
    #   ipyparallel
    #   ipywidgets
ipyparallel==8.4.1
    # via hyperspy
ipython==7.34.0
    # via
    #   hyperspy
    #   ipykernel
    #   ipyparallel
    #   ipywidgets
ipywidgets==8.0.6
    # via pyxem
isort==5.10.1
    # via pylint
jedi==0.18.1
    # via ipython
jinja2==3.1.2
    # via hyperspy
joblib==1.2.0
    # via scikit-learn
jupyter-client==7.3.5
    # via
    #   ipykernel
    #   ipyparallel
jupyter-core==5.1.5
    # via jupyter-client
jupyterlab-widgets==3.0.7
    # via ipywidgets
kikuchipy==0.8.4
    # via pynxtools (pyproject.toml)
kiwisolver==1.4.4
    # via matplotlib
lark==1.1.5
    # via pynxtools (pyproject.toml)
lazy-object-proxy==1.9.0
    # via astroid
llvmlite==0.39.1
    # via numba
lmfit==1.2.0
    # via pyxem
locket==1.0.0
    # via partd
markupsafe==2.1.1
    # via jinja2
matplotlib==3.5.3
    # via
    #   ase
    #   diffsims
    #   hyperspy
    #   kikuchipy
    #   matplotlib-scalebar
    #   orix
    #   pyfai
    #   pyxem
    #   radioactivedecay
matplotlib-inline==0.1.6
    # via
    #   ipykernel
    #   ipython
matplotlib-scalebar==0.8.1
    # via orix
mccabe==0.7.0
    # via pylint
mergedeep==1.3.4
    # via pynxtools (pyproject.toml)
mpmath==1.2.1
    # via sympy
mypy==1.2.0
    # via pynxtools (pyproject.toml)
mypy-extensions==1.0.0
    # via mypy
nanonispy @ git+https://github.com/ramav87/nanonispy.git@a0da87c58482d29624a2bf5deecb763dd1274212
    # via pynxtools (pyproject.toml)
natsort==8.2.0
    # via hyperspy
nest-asyncio==1.5.6
    # via
    #   ipykernel
    #   jupyter-client
networkx==2.6.3
    # via
    #   radioactivedecay
    #   scikit-image
niondata==0.15.3
    # via
    #   nionswift
    #   nionswift-io
nionswift==0.16.8
    # via pynxtools (pyproject.toml)
nionswift-io==0.15.1
    # via nionswift
nionui==0.6.10
    # via nionswift
nionutils==0.4.6
    # via
    #   niondata
    #   nionswift
    #   nionswift-io
    #   nionui
numba==0.56.2
    # via
    #   diffsims
    #   hyperspy
    #   kikuchipy
    #   orix
    #   pyxem
    #   sparse
numcodecs==0.10.2
    # via zarr
numexpr==2.8.3
    # via
    #   hyperspy
    #   pyfai
numpy==1.21.6
    # via
    #   ase
    #   dask
    #   diffsims
    #   fabio
    #   h5py
    #   hyperspy
    #   ifes-apt-tc-data-modeling
    #   imageio
    #   kikuchipy
    #   lmfit
    #   matplotlib
    #   nanonispy
    #   niondata
    #   nionswift
    #   nionswift-io
    #   nionui
    #   numba
    #   numcodecs
    #   numexpr
    #   numpy-quaternion
    #   orix
    #   pandas
    #   pyfai
    #   pynxtools (pyproject.toml)
    #   pywavelets
    #   pyxem
    #   radioactivedecay
    #   scikit-image
    #   scikit-learn
    #   scipy
    #   silx
    #   sparse
    #   tifffile
    #   xarray
    #   zarr
numpy-quaternion==2022.4.3
    # via orix
orix==0.11.1
    # via
    #   diffsims
    #   kikuchipy
    #   pyxem
packaging==21.3
    # via
    #   build
    #   dask
    #   hyperspy
    #   ipykernel
    #   matplotlib
    #   numexpr
    #   pint
    #   pooch
    #   pytest
    #   scikit-image
pandas==1.3.5
    # via
    #   ifes-apt-tc-data-modeling
    #   pynxtools (pyproject.toml)
    #   xarray
parso==0.8.3
    # via jedi
partd==1.3.0
    # via dask
pexpect==4.8.0
    # via ipython
pickleshare==0.7.5
    # via ipython
pillow==9.2.0
    # via
    #   imageio
    #   matplotlib
    #   nionswift
    #   scikit-image
pint==0.18
    # via hyperspy
pip-tools==6.13.0
    # via pynxtools (pyproject.toml)
platformdirs==2.5.2
    # via
    #   jupyter-core
    #   pooch
    #   pylint
pluggy==1.0.0
    # via pytest
pooch==1.7.0
    # via
    #   kikuchipy
    #   orix
prettytable==3.4.1
    # via hyperspy
prompt-toolkit==3.0.31
    # via ipython
psutil==5.9.2
    # via
    #   diffsims
    #   ipykernel
    #   ipyparallel
    #   pyxem
ptyprocess==0.7.0
    # via pexpect
pycifrw==4.4.5
    # via diffpy-structure
pycodestyle==2.9.1
    # via pynxtools (pyproject.toml)
pyfai==2023.3.0
    # via pyxem
pygments==2.13.0
    # via ipython
pylint==2.17.2
    # via pynxtools (pyproject.toml)
pyparsing==3.0.9
    # via
    #   matplotlib
    #   packaging
pyproject-hooks==1.0.0
    # via build
pytest==7.3.1
    # via
    #   pynxtools (pyproject.toml)
    #   pytest-cov
    #   pytest-timeout
pytest-cov==4.0.0
    # via pynxtools (pyproject.toml)
pytest-timeout==2.1.0
    # via pynxtools (pyproject.toml)
python-dateutil==2.8.2
    # via
    #   hyperspy
    #   ipyparallel
    #   jupyter-client
    #   matplotlib
    #   pandas
pytz==2022.4
    # via
    #   nionswift
    #   pandas
    #   pynxtools (pyproject.toml)
pytz-deprecation-shim==0.1.0.post0
    # via tzlocal
pywavelets==1.3.0
    # via scikit-image
pyxem==0.15.0
    # via pynxtools (pyproject.toml)
pyyaml==6.0
    # via
    #   dask
    #   hyperspy
    #   kikuchipy
    #   pynxtools (pyproject.toml)
pyzmq==24.0.1
    # via
    #   ipykernel
    #   ipyparallel
    #   jupyter-client
radioactivedecay==0.4.16
    # via ifes-apt-tc-data-modeling
requests==2.28.1
    # via
    #   hyperspy
    #   pooch
    #   pynxtools (pyproject.toml)
    #   requests-cache
requests-cache==0.9.8
    # via pynxtools (pyproject.toml)
scikit-image==0.19.3
    # via
    #   hyperspy
    #   kikuchipy
    #   pyxem
scikit-learn==1.2.2
    # via
    #   kikuchipy
    #   pyxem
scipy==1.7.3
    # via
    #   ase
    #   diffsims
    #   hyperspy
    #   kikuchipy
    #   lmfit
    #   niondata
    #   nionswift
    #   orix
    #   pyfai
    #   pynxtools (pyproject.toml)
    #   pyxem
    #   radioactivedecay
    #   scikit-image
    #   scikit-learn
    #   sparse
silx==1.1.2
    # via pyfai
six==1.16.0
    # via
    #   diffpy-structure
    #   python-dateutil
    #   url-normalize
smmap==5.0.0
    # via gitdb
sparse==0.13.0
    # via hyperspy
structlog==23.1.0
    # via pynxtools (pyproject.toml)
sympy==1.10.1
    # via
    #   hyperspy
    #   radioactivedecay
threadpoolctl==3.1.0
    # via scikit-learn
tifffile==2021.11.2
    # via
    #   hyperspy
    #   scikit-image
tomli==2.0.1
    # via
    #   build
    #   coverage
    #   mypy
    #   pylint
    #   pyproject-hooks
    #   pytest
tomlkit==0.11.7
    # via pylint
toolz==0.12.0
    # via
    #   dask
    #   hyperspy
    #   partd
tornado==6.2
    # via
    #   ipykernel
    #   ipyparallel
    #   jupyter-client
tqdm==4.64.1
    # via
    #   diffsims
    #   hyperspy
    #   ipyparallel
    #   kikuchipy
    #   orix
traitlets==5.4.0
    # via
    #   ipykernel
    #   ipyparallel
    #   ipython
    #   ipywidgets
    #   jupyter-client
    #   jupyter-core
    #   matplotlib-inline
traits==6.4.1
    # via hyperspy
transforms3d==0.4.1
    # via
    #   diffsims
    #   pyxem
types-pytz==2022.6.0.1
    # via pynxtools (pyproject.toml)
types-pyyaml==6.0.12
    # via pynxtools (pyproject.toml)
types-requests==2.28.11.12
    # via pynxtools (pyproject.toml)
types-urllib3==1.26.25.5
    # via types-requests
typing-extensions==4.3.0
    # via
    #   astroid
    #   mypy
    #   numcodecs
tzdata==2023.3
    # via pytz-deprecation-shim
tzlocal==4.3
    # via
    #   nionswift
    #   pynxtools (pyproject.toml)
uncertainties==3.1.7
    # via lmfit
url-normalize==1.4.3
    # via requests-cache
urllib3==1.26.12
    # via
    #   requests
    #   requests-cache
wcwidth==0.2.5
    # via
    #   prettytable
    #   prompt-toolkit
wheel==0.40.0
    # via pip-tools
widgetsnbextension==4.0.7
    # via ipywidgets
wrapt==1.14.1
    # via astroid
xarray==0.20.2
    # via pynxtools (pyproject.toml)
zarr==2.12.0
    # via hyperspy
zipfile37==0.1.3
    # via pynxtools (pyproject.toml)
zipp==3.8.1
    # via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# pip
# setuptools
dkriegner commented 9 months ago

Thanks for your report. I can not reproduce this since in the xrayutilities code there is no direct import of numpy.core.multiarray which seems to cause your issue. Also Python 3.8 is covered by the CI tests on azure where it works just fine.

So it must be a problem in one of the dependencies or your installation. I think it could be the latter one.

Few questions: 1) What xrayutilities version do you use? It's not listed in the package list you sent. 2) How did you install xrayutilities?

My feeling is that it could be a problem of the binary blob in xrayutilities being linked to a different numpy version as compared to the one you are using. This could mean the binary blobs which are distributed via pypi do not work for you. Maybe installing xrayutilities from sources would fix the issue in this case.

RubelMozumder commented 9 months ago

Thank you for your response. Sorry, for missing xrayutilities version.

  1. Version of the x-ray utilities version: 1.7.4
  2. This was added to the dependency list pyproject later installed with pip package manager. While creating dev-requirments.txt from pyproject I did not get any version conflict with numpy lib.

One important note: The error ImportError: numpy.core.multiarray failed to import can not be even reproduced in my local Python installation. The error I sheared in this issue comes from GitHub ci/cd workflow though the same workflow in passing without any issue in local installation.

dkriegner commented 9 months ago

As you describe it, this sounds to me is unlikely to be causes by a code problem inside xrayutilities. It could potentially be related to the binary package on PyPI. But since as you yourself write it's difficult to reproduce this problem, I am not sure how to tackle it.

I would certainly need you to debug this more to identify the problem. Also is the problem limited to Python 3.8?

One more hint: On PyPI I see that the numpy-1.21.6 version for python 3.8 is distributed only as manylinux 2010 version whereas xrayutilities-1.7.4 is distributed as manylinux 2014 variant. not sure if this can cause an issue. In your local pipeline do you use the very same package versions? I also do not fully get why the numpy version used by your pipeline is more than a year old.