search

bjodah / pycvodes

Python wrapper around cvodes (from the sundials library)
BSD 2-Clause "Simplified" License
35 stars 5 forks source link

pytest error: cannot open shared object file: No such file or directory #138

Closed Whitechapel-1888 closed 6 months ago

Whitechapel-1888 commented 7 months ago

Short Problem description: running pytest gives an error despite installing pycvodes as instructed on the readme.

Detailed description: On a fresh & empty conda env, running the command conda install -c conda-forge pycvodes pytest works flawlessly, but running python -m pytest --pyargs pycvodes throws the following error:

python -m pytest --pyargs pycvodes
===================================================================== test session starts ======================================================================
platform linux -- Python 3.12.1, pytest-8.0.0, pluggy-1.4.0
rootdir: /home/USER
collected 0 items / 1 error

============================================================================ ERRORS ============================================================================
________________________ ERROR collecting source/miniconda3/envs/cyth/lib/python3.12/site-packages/pycvodes/tests/test_cvodes_numpy.py _________________________
ImportError while importing test module '/home/USER/source/miniconda3/envs/cyth/lib/python3.12/site-packages/pycvodes/tests/test_cvodes_numpy.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
source/miniconda3/envs/cyth/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
source/miniconda3/envs/cyth/lib/python3.12/site-packages/pycvodes/__init__.py:9: in <module>
    from ._cvodes import (
E   ImportError: libsundials_sunlinsollapackdense.so.4.6.1: cannot open shared object file: No such file or directory
=================================================================== short test summary info ====================================================================
ERROR source/miniconda3/envs/cyth/lib/python3.12/site-packages/pycvodes/tests/test_cvodes_numpy.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================= 1 error in 0.40s =======================================================================
bjodah commented 7 months ago

Thank you for reporting this. I don't use conda-forge myself, so issues like these might go unnoticed by me for a while.

It looks like tests did work for at least on python-3.12 build. Maybe it's enough to trigger a rerender of the recipe over at conda-forge.

Whitechapel-1888 commented 7 months ago

Thank you for reporting this. I don't use conda-forge myself, so issues like these might go unnoticed by me for a while.

It looks like tests did work for at least on python-3.12 build. Maybe it's enough to trigger a rerender of the recipe over at conda-forge.

So there may be an inconsistency in the version on conda-forge? That is the recommended installation method however. I assume you build it locally?

On my env, conda installed python-3.12.1 and cython-3.0.8. Install output:

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2024.2.2             hbcca054_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
cython                    3.0.8           py312h5eee18b_0
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
expat                     2.5.0                h6a678d5_0
gmp                       6.3.0                h59595ed_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ld_impl_linux-64          2.38                 h1181459_1
libblas                   3.9.0           21_linux64_openblas    conda-forge
libcblas                  3.9.0           21_linux64_openblas    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.4                h6a678d5_0
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgfortran-ng            13.2.0               h69a702a_5    conda-forge
libgfortran5              13.2.0               ha4646dd_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libhwloc                  2.9.3           default_h554bfaf_1009    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           21_linux64_openblas    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
libsqlite                 3.45.1               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.5               h232c23b_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
metis                     5.1.0             h59595ed_1007    conda-forge
mpfr                      4.2.1                h9458935_0    conda-forge
ncurses                   6.4                  h6a678d5_0
numpy                     1.26.4          py312heda63a1_0    conda-forge
openssl                   3.2.1                hd590300_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pip                       23.3.1          py312h06a4308_0
pluggy                    1.4.0              pyhd8ed1ab_0    conda-forge
pycvodes                  0.14.2          py312he60df36_2    conda-forge
pytest                    8.0.0              pyhd8ed1ab_0    conda-forge
python                    3.12.1          hab00c5b_1_cpython    conda-forge
python_abi                3.12                    4_cp312    conda-forge
readline                  8.2                  h5eee18b_0
setuptools                68.2.2          py312h06a4308_0
sqlite                    3.41.2               h5eee18b_0
suitesparse               5.10.1               h5a4f163_3    conda-forge
sundials                  6.6.2                h777d08e_1    conda-forge
tbb                       2021.11.0            h00ab1b0_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2023d                h04d1e81_0
wheel                     0.41.2          py312h06a4308_0
xz                        5.4.5                h5eee18b_0
zlib                      1.2.13               hd590300_5    conda-forge
bjodah commented 7 months ago

Thanks. Right, maybe I should remove those instructions (or try to find more maintainers of the recipe, we will see). Yes, I'm thinking there's some inconsistency that wasn't detected automatically over at conda-forge. That output, does it signify a failure or success? (I'm currently on a chrome book)

Whitechapel-1888 commented 7 months ago

Sorry for my late reply.

That output, does it signify a failure or success? (I'm currently on a chrome book)

This is the list of successfully installed python libraries. Installation works (as far as I can tell).

moorepants commented 6 months ago

I tried reproducing but do not get the same error:

$ conda create -n pycvodes pycvodes pytest
Retrieving notices: ...working... done
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
    current version: 23.11.0
    latest version: 24.1.1

Please update conda by running

    $ conda update -n base -c conda-forge conda

## Package Plan ##

  environment location: /home/moorepants/miniconda/envs/pycvodes

  added / updated specs:
    - pycvodes
    - pytest

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |      conda_forge           3 KB  conda-forge
    _openmp_mutex-4.5          |            2_gnu          23 KB  conda-forge
    libblas-3.9.0              |21_linux64_openblas          14 KB  conda-forge
    libcblas-3.9.0             |21_linux64_openblas          14 KB  conda-forge
    libgfortran-ng-13.2.0      |       h69a702a_5          23 KB  conda-forge
    liblapack-3.9.0            |21_linux64_openblas          14 KB  conda-forge
    numpy-1.26.4               |  py312heda63a1_0         7.1 MB  conda-forge
    pycvodes-0.14.2            |  py312he60df36_3         400 KB  conda-forge
    python-3.12.2              |hab00c5b_0_cpython        30.8 MB  conda-forge
    python_abi-3.12            |          4_cp312           6 KB  conda-forge
    setuptools-69.1.0          |     pyhd8ed1ab_1         459 KB  conda-forge
    sundials-6.7.0             |       h777d08e_0         935 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        39.8 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu 
  bzip2              conda-forge/linux-64::bzip2-1.0.8-hd590300_5 
  ca-certificates    conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0 
  colorama           conda-forge/noarch::colorama-0.4.6-pyhd8ed1ab_0 
  exceptiongroup     conda-forge/noarch::exceptiongroup-1.2.0-pyhd8ed1ab_2 
  gmp                conda-forge/linux-64::gmp-6.3.0-h59595ed_0 
  icu                conda-forge/linux-64::icu-73.2-h59595ed_0 
  iniconfig          conda-forge/noarch::iniconfig-2.0.0-pyhd8ed1ab_0 
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0 
  libblas            conda-forge/linux-64::libblas-3.9.0-21_linux64_openblas 
  libcblas           conda-forge/linux-64::libcblas-3.9.0-21_linux64_openblas 
  libexpat           conda-forge/linux-64::libexpat-2.5.0-hcb278e6_1 
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5 
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_5 
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-13.2.0-h69a702a_5 
  libgfortran5       conda-forge/linux-64::libgfortran5-13.2.0-ha4646dd_5 
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h807b86a_5 
  libhwloc           conda-forge/linux-64::libhwloc-2.9.3-default_h554bfaf_1009 
  libiconv           conda-forge/linux-64::libiconv-1.17-hd590300_2 
  liblapack          conda-forge/linux-64::liblapack-3.9.0-21_linux64_openblas 
  libnsl             conda-forge/linux-64::libnsl-2.0.1-hd590300_0 
  libopenblas        conda-forge/linux-64::libopenblas-0.3.26-pthreads_h413a1c8_0 
  libsqlite          conda-forge/linux-64::libsqlite-3.45.1-h2797004_0 
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.2.0-h7e041cc_5 
  libuuid            conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0 
  libxcrypt          conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1 
  libxml2            conda-forge/linux-64::libxml2-2.12.5-h232c23b_0 
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5 
  metis              conda-forge/linux-64::metis-5.1.0-h59595ed_1007 
  mpfr               conda-forge/linux-64::mpfr-4.2.1-h9458935_0 
  ncurses            conda-forge/linux-64::ncurses-6.4-h59595ed_2 
  numpy              conda-forge/linux-64::numpy-1.26.4-py312heda63a1_0 
  openssl            conda-forge/linux-64::openssl-3.2.1-hd590300_0 
  packaging          conda-forge/noarch::packaging-23.2-pyhd8ed1ab_0 
  pip                conda-forge/noarch::pip-24.0-pyhd8ed1ab_0 
  pluggy             conda-forge/noarch::pluggy-1.4.0-pyhd8ed1ab_0 
  pycvodes           conda-forge/linux-64::pycvodes-0.14.2-py312he60df36_3 
  pytest             conda-forge/noarch::pytest-8.0.0-pyhd8ed1ab_0 
  python             conda-forge/linux-64::python-3.12.2-hab00c5b_0_cpython 
  python_abi         conda-forge/linux-64::python_abi-3.12-4_cp312 
  readline           conda-forge/linux-64::readline-8.2-h8228510_1 
  setuptools         conda-forge/noarch::setuptools-69.1.0-pyhd8ed1ab_1 
  suitesparse        conda-forge/linux-64::suitesparse-5.10.1-h5a4f163_3 
  sundials           conda-forge/linux-64::sundials-6.7.0-h777d08e_0 
  tbb                conda-forge/linux-64::tbb-2021.11.0-h00ab1b0_1 
  tk                 conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101 
  tomli              conda-forge/noarch::tomli-2.0.1-pyhd8ed1ab_0 
  tzdata             conda-forge/noarch::tzdata-2024a-h0c530f3_0 
  wheel              conda-forge/noarch::wheel-0.42.0-pyhd8ed1ab_0 
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0 

Downloading and Extracting Packages:

Preparing transaction: done                                                                                                                                                                   
Verifying transaction: done                                                                                                                                                                   
Executing transaction: done                                                                                                                                                                   
#                                                                                                                                                                                             
# To activate this environment, use                                                                                                                                                           
#                                                                                                                                                                                             
#     $ conda activate pycvodes                                                                                                                                                               
#                                                                                                                                                                                             
# To deactivate an active environment, use                                                                                                                                                    
#                                                                                                                                                                                             
#     $ conda deactivate                                                                                                                                                                      

$ act pycvodes 
(pycvodes) $  python -m pytest --pyargs pycvodes
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0
rootdir: /home/moorepants/Manuscripts/learn-multibody-dynamics
collected 55 items                                                                                                                                                                           

tests/test_cvodes_numpy.py ....................................................s..                                                                                                     [100%]

====================================================================================== warnings summary ======================================================================================
tests/test_cvodes_numpy.py::test_get_include
  /home/moorepants/miniconda/envs/pycvodes/lib/python3.12/site-packages/pycvodes/__init__.py:17: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import resource_filename, Requirement

tests/test_cvodes_numpy.py: 75 warnings
  /home/moorepants/miniconda/envs/pycvodes/lib/python3.12/site-packages/pycvodes/_util.py:12: DeprecationWarning: NotImplemented should not be used in a boolean context
    if any(map(set(nonnull_opts).__eq__, valid_arg_combs)):

tests/test_cvodes_numpy.py::test_integrate_predefined[bdf-10-False]
tests/test_cvodes_numpy.py::test_integrate_predefined[bdf-10-True]
tests/test_cvodes_numpy.py::test_jtimes_predefined[gmres-False]
tests/test_cvodes_numpy.py::test_jtimes_predefined[gmres_classic-False]
tests/test_cvodes_numpy.py::test_jtimes_predefined[bicgstab-False]
tests/test_cvodes_numpy.py::test_jtimes_predefined[tfqmr-False]
  /home/moorepants/miniconda/envs/pycvodes/lib/python3.12/site-packages/pycvodes/__init__.py:280: UserWarning: No full jacobian provided; disabling default preconditioning.
    return predefined(

tests/test_cvodes_numpy.py::test_integrate_predefined[bdf-10-False]
tests/test_cvodes_numpy.py::test_integrate_predefined[bdf-10-True]
  /home/moorepants/miniconda/envs/pycvodes/lib/python3.12/site-packages/pycvodes/__init__.py:280: UserWarning: Method requires jacobian, no callback provided: using finite differences (may be inaccurate).
    return predefined(

tests/test_cvodes_numpy.py::test_jtimes_adaptive[gmres-False]
tests/test_cvodes_numpy.py::test_jtimes_adaptive[gmres_classic-False]
tests/test_cvodes_numpy.py::test_jtimes_adaptive[bicgstab-False]
tests/test_cvodes_numpy.py::test_jtimes_adaptive[tfqmr-False]
  /home/moorepants/miniconda/envs/pycvodes/lib/python3.12/site-packages/pycvodes/__init__.py:150: UserWarning: No full jacobian provided; disabling default preconditioning.
    return adaptive(rhs, jac, np.ascontiguousarray(y0, dtype=np.float64), x0, xend,

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= 54 passed, 1 skipped, 88 warnings in 0.40s =========================================================================
bjodah commented 6 months ago

Thank you Jason. @Whitechapel-1888 what platform are you on?

Whitechapel-1888 commented 6 months ago

I tried reproducing but do not get the same error:

Hmm, I installed everything from conda-forge, maybe that is the issue. I will try again, but this has to wait until tomorrow.

@Whitechapel-1888 what platform are you on?

Debian on wsl2:

       └─ $ uname -a
Linux ouroboros 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 GNU/Linux
moorepants commented 6 months ago

Hmm, I installed everything from conda-forge, maybe that is the issue. I will try again, but this has to wait until tomorrow.

The conda list output you show above seems to have a mix of conda forge and something else (blank).

Whitechapel-1888 commented 6 months ago

The conda list output you show above seems to have a mix of conda forge and something else (blank).

Those libraries with blank fields were installed from the default conda resource

bjodah commented 6 months ago

@Whitechapel-1888 that explains it. Look at the version numbers of sundials: 4 vs 6. Those are not compatible I'm afraid. I don't know how channel priorities are handled in conda, but if our README needs updating, or if our conda recipe needs updating, we should see to it that it is fixed.

moorepants commented 6 months ago

You would need the priority set to strict so that everything is grabbed from conda forge (unless it doesn't exist there). Also best practice may be to install in a conda env with only conda forge as a source.

Whitechapel-1888 commented 6 months ago

Unfortunately, the workload this week is mental. I just managed to claw back some spare time to test things...

You would need the priority set to strict so that everything is grabbed from conda forge (unless it doesn't exist there). Also best practice may be to install in a conda env with only conda forge as a source.

I added conda-forge as the highest priority channel in conda and then installed pycvodes exactly as you did and it works. I am getting 88 warnings like you, but I assume these can be ignored. Thanks for helping out!

I would recommend updating the installation guide by simply adding that all necessary packages should be pulled from conda-forge only. conda config --add channels conda-forge will do the trick globally.

moorepants commented 6 months ago

This is the standard information for using conda forge packages: https://conda-forge.org/docs/user/introduction/

We can add a note about this but I'm not sure we need to explain this. Same for recommending various options for pip.