Linux library paths for libnd2readsdk-shared.so not set correctly on install

[VolkerH]

• nd2 version: latest main
• Python version: 3.9
• Operating System: Ubuntu 20.04

Description

When pulling the latest main and doing a local install with pip install . or pip install -e . from the nd2 folder the shared library doesn't get installed correctly. When executing code from the location of where I installed the repo, everything works fine. When executing the code from a different working directory I get import errors.

What I Did

running my tests from the nd2 directory works without ImportError (note that the Python Kernel dies when reading nd2, I am investigating and will open a separate issue).

(napari_latest) hilsenst@itservices-XPS-15-9500:~/Github/nd2$ pytest ~/GitlabEMBL/spacem-ht/src/spacem-mosaic/tests/test_io.py 
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
PyQt5 5.15.4 -- Qt runtime 5.15.2 -- Qt compiled 5.15.2
rootdir: /home/hilsenst/GitlabEMBL/spacem-ht/src/spacem-mosaic
plugins: order-1.0.0, napari-0.4.11, timeout-1.4.2, anyio-3.3.0, napari-plugin-engine-0.1.9, qt-4.0.2, hypothesis-6.14.4
collected 4 items                                                                                                                                                                                         

../../GitlabEMBL/spacem-ht/src/spacem-mosaic/tests/test_io.py ...Killed

running my tests from some other directory fails with ImportError:

(napari_latest) hilsenst@itservices-XPS-15-9500:~/Github$ pytest ~/GitlabEMBL/spacem-ht/src/spacem-mosaic/tests/test_io.py 
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
PyQt5 5.15.4 -- Qt runtime 5.15.2 -- Qt compiled 5.15.2
rootdir: /home/hilsenst/GitlabEMBL/spacem-ht/src/spacem-mosaic
plugins: order-1.0.0, napari-0.4.11, timeout-1.4.2, anyio-3.3.0, napari-plugin-engine-0.1.9, qt-4.0.2, hypothesis-6.14.4
collected 4 items                                                                                                                                                                                         

../GitlabEMBL/spacem-ht/src/spacem-mosaic/tests/test_io.py ...F                                                                                                                                     [100%]

================================================================================================ FAILURES =================================================================================================
___________________________________________________________________________________________ test_load_tiles_nd2 ___________________________________________________________________________________________

    def test_load_tiles_nd2():
>       tile_arr, channel_dict = load_tiles(Path(dataset_nd2))

../GitlabEMBL/spacem-ht/src/spacem-mosaic/tests/test_io.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../GitlabEMBL/spacem-ht/src/spacem-mosaic/spacem_mosaic/io.py:110: in load_tiles
    return load_tiles_nd2(file_path)
../GitlabEMBL/spacem-ht/src/spacem-mosaic/spacem_mosaic/io.py:54: in load_tiles_nd2
    f = ND2File(file_path)
../miniconda3/envs/napari_latest/lib/python3.9/site-packages/nd2/nd2file.py:42: in __init__
    self._rdr = get_reader(self._path)
../miniconda3/envs/napari_latest/lib/python3.9/site-packages/nd2/_util.py:24: in get_reader
    from ._sdk.latest import ND2Reader
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   from . import latest
E   ImportError: libnd2readsdk-shared.so: cannot open shared object file: No such file or directory

../miniconda3/envs/napari_latest/lib/python3.9/site-packages/nd2/_sdk/__init__.py:1: ImportError
============================================================================================ warnings summary =============================================================================================
../miniconda3/envs/napari_latest/lib/python3.9/site-packages/pims/cine.py:29
  /home/hilsenst/miniconda3/envs/napari_latest/lib/python3.9/site-packages/pims/cine.py:29: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    from collections import Iterable

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================= short test summary info =========================================================================================
FAILED ../GitlabEMBL/spacem-ht/src/spacem-mosaic/tests/test_io.py::test_load_tiles_nd2 - ImportError: libnd2readsdk-shared.so: cannot open shared object file: No such file or directory
================================================================================= 1 failed, 3 passed, 1 warning in 8.65s ==================================================================================
(napari_latest) hilsenst@itservices-XPS-15-9500:~/Github$ cd nd2
(napari_latest) hilsenst@itservices-XPS-15-9500:~/Github/nd2$ pytest ~/GitlabEMBL/spacem-ht/src/spacem-mosaic/tests/test_io.py 
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1

I suspect that the issue is somewhere here:

https://github.com/tlambert03/nd2/blob/main/setup.py#L10-L13

But haven't been able to figure out exactly what goes wrong.

[tlambert03]

This is tricky. For the wheel on pypi, cibuildwheel takes care of all the relocatability issues for us... but that doesn't happen when building locally. It's "enough" for tests on CI but not super robust. Will look into it

[VolkerH]

As a workaround, I .resolve()d the relative paths in setup.py and install with pip -e. That is not working with just pip so is not the correct way, but it lets me install local changes into my python environment.

[tlambert03]

@shenker was able to build locally on a linux using the following:

mamba env create -n nd2test
mamba activate nd2test
# installing deps from conda... not sure if this part is critical 
mamba install -c conda-forge cython black flake8 flake8-docstrings \
    imagecodecs aicsimageio ipython isort mypy pre-commit psutil \
    pydocstyle pytest pytest-cov wurlitzer xarray resource_backed_dask_array \
    python=3.10.4 requests
git clone git@github.com:tlambert03/nd2.git
cd nd2
python scripts/download_samples.py
make
pip install --no-deps -e .
# i bet this part is critical ... since the wheel isn't relocatable unless you build with cibuildwheel
LD_LIBRARY_PATH=$CONDA_PREFIX/lib pytest

Since this is ultimately doesn't affect the way the package gets built and distributed on linux for either pypi or conda forge, I'm going to close this as difficult to fix. If you try again, let me know if setting the LD_LIBRARY_PATH solves it in your local dev environment

[uthapa82]

Facing Same issue as @VolkerH , ubuntu 20.04

[tlambert03]

sorry about that @uthapa82. Are you trying to install for local development? Did you try @shenker's approach in https://github.com/tlambert03/nd2/issues/24#issuecomment-1147923768?

[uthapa82]

@tlambert03 No worries, it happens. I am just trying to use it to convert nd2 to jpeg file, but can't seem to use the module. I tried adding LD_LIBRARY_PATH no luck there, I tried @shenker's approach that also didn't worked form me, for now I used the windows VM and seems to work perfectly there but not on Ubuntu 20.04

[tlambert03]

If you're trying to just use the library, how did you install? Installation from conda is your best bet

[uthapa82]

Sound good, I am just trying to use to it. So probably will try with conda and see if that solves. Appreciate your information