search

mobie / mobie-utils-python

Python tools for MoBIE
MIT License
9 stars 5 forks source link

make package available in PyPI #49

Closed martinschorb closed 2 years ago

martinschorb commented 2 years ago

Hi,

I am running a CI job to test one of my packages that depends on mobie. I'd like to install mobie from a dependencies list using pip.

Would it be possible to make it available on PyPI?

martinschorb commented 2 years ago

same holds for https://github.com/constantinpape/pybdv/

Although there it seems that PyPI provides an old version.

constantinpape commented 2 years ago

Unfortunately this is not currently possible for mobie-utils-python, since it has several dependencies that are not available via PyPI and that are very difficult to provide through it in a reliable fashion:

These are c++ packages with python bindings, for which building wheels is not straight forward. There are some efforts to enable building wheels from conda recipes, e.g. https://github.com/conda-incubator/conda-press. But I don't know how well this would work in these cases and I don't have time to look into this.

same holds for https://github.com/constantinpape/pybdv/

That's possible since it does not rely on any upstream dependencies that are not available on PyPI.

Although there it seems that PyPI provides an old version.

Really? Could you link to it? I couldn't find it; and it's not by me.

I am running a CI job to test one of my packages that depends on mobie.

If you're using GH Actions then using conda is very easy: https://github.com/marketplace/actions/setup-miniconda

martinschorb commented 2 years ago

I am using Gitlab CI, because I need to make some custom-built containers available as web service APIs to run my tests against.

I will try to base my test run on anacoda then and get my dependencies through there.

I think I had an older version of pybdv installed in that environment, so when running pip install it did simply fetch the local installation. In a fresh clean environment it does not find it, so false alarm...

martinschorb commented 2 years ago

I got my conda container up and running. I am now running into a strange error when importing mobie:

import mobie
/opt/conda/lib/python3.7/site-packages/mobie/__init__.py:1: in <module>
    from .image_data import add_image, add_bdv_image
/opt/conda/lib/python3.7/site-packages/mobie/image_data.py:5: in <module>
    import mobie.metadata as metadata
/opt/conda/lib/python3.7/site-packages/mobie/metadata/__init__.py:1: in <module>
    from .bookmark_metadata import add_additional_bookmark, add_dataset_bookmark, add_grid_bookmark
/opt/conda/lib/python3.7/site-packages/mobie/metadata/bookmark_metadata.py:6: in <module>
    from .view_metadata import get_view, get_grid_view
/opt/conda/lib/python3.7/site-packages/mobie/metadata/view_metadata.py:8: in <module>
    from ..tables import check_source_annotation_table, compute_source_annotation_table
/opt/conda/lib/python3.7/site-packages/mobie/tables/__init__.py:3: in <module>
    from .traces_table import compute_trace_default_table
/opt/conda/lib/python3.7/site-packages/mobie/tables/traces_table.py:11: in <module>
    from ..import_data.traces import parse_traces, vals_to_coords
/opt/conda/lib/python3.7/site-packages/mobie/import_data/__init__.py:5: in <module>
    from .traces import import_traces
/opt/conda/lib/python3.7/site-packages/mobie/import_data/traces.py:5: in <module>
    import elf.skeleton.io as skio
/opt/conda/lib/python3.7/site-packages/elf/skeleton/__init__.py:1: in <module>
    from .skeletonize import skeletonize, get_method_names
/opt/conda/lib/python3.7/site-packages/elf/skeleton/skeletonize.py:1: in <module>
    from .thinning import thinning
/opt/conda/lib/python3.7/site-packages/elf/skeleton/thinning.py:3: in <module>
    from skan import csr
/opt/conda/lib/python3.7/site-packages/skan/__init__.py:1: in <module>
    from .csr import skeleton_to_csgraph, branch_statistics, summarize, Skeleton
/opt/conda/lib/python3.7/site-packages/skan/csr.py:91: in <module>
    @numba.jit(nopython=True, cache=True, nogil=True)
/opt/conda/lib/python3.7/site-packages/numba/core/decorators.py:212: in wrapper
    disp.enable_caching()
/opt/conda/lib/python3.7/site-packages/numba/core/dispatcher.py:863: in enable_caching
    self._cache = FunctionCache(self.py_func)
/opt/conda/lib/python3.7/site-packages/numba/core/caching.py:613: in __init__
    self._impl = self._impl_class(py_func)
/opt/conda/lib/python3.7/site-packages/numba/core/caching.py:351: in __init__
    "for file %r" % (qualname, source_path))
E   RuntimeError: cannot cache function '_write_pixel_graph': no locator available for file '/opt/conda/lib/python3.7/site-packages/skan/csr.py'

Any idea what could cause this?

constantinpape commented 2 years ago

Hmm, this looks like a numba error caused by the skan import. Have you seen this one before @jni?

@martinschorb, can you try this with a more recent python version? 3.7 is fairly old, maybe that could fix it. If not I could update elf, so that skan only gets imported on demand.

martinschorb commented 2 years ago

unfortunately, one of the core packages I am using still relies on 3.7, so at the moment I am stuck using that.

martinschorb commented 2 years ago

Also, it runs well on 3.7 in my other environments (Mac, Linux). Just inside the continuumio/miniconda3 container, it refuses to import after installing everything.

constantinpape commented 2 years ago

Can you run a conda list in that container and paste the output here?

martinschorb commented 2 years ago

The issue arises when running python as a non-privileged user inside the container.

edit: minimal example only installing mobie_utils from conda-forge, nothing else ...

# su testuser1 -c 'conda list'
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
affogato                  0.3.3            py37hb3b4854_0    conda-forge
aiobotocore               2.2.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.1            py37h5e8e339_0    conda-forge
aioitertools              0.10.0             pyhd8ed1ab_0    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
asynctest                 0.13.0                     py_0    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
blosc                     1.21.0               h9c3ff4c_0    conda-forge
boost                     1.74.0           py37h796e4cb_5    conda-forge
boost-cpp                 1.74.0               h6cacc03_7    conda-forge
botocore                  1.24.21            pyhd8ed1ab_1    conda-forge
brotlipy                  0.7.0           py37h27cfd23_1003  
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
certifi                   2021.10.8        py37h89c1867_1    conda-forge
cffi                      1.15.0           py37hd667e15_1  
chardet                   4.0.0           py37h06a4308_1003  
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
cloudpickle               2.0.0              pyhd8ed1ab_0    conda-forge
cluster_tools             0.4.1              pyhd8ed1ab_0    conda-forge
coin-or-cbc               2.10.7               h3786ebc_0    conda-forge
coin-or-cgl               0.60.3               he7e83c3_2    conda-forge
coin-or-clp               1.17.6               h256e9bb_3    conda-forge
coin-or-osi               0.108.6              h3b589db_2    conda-forge
coin-or-utils             2.11.6               h573740c_0    conda-forge
coincbc                   2.10.7            0_metapackage    conda-forge
conda                     4.12.0           py37h89c1867_0    conda-forge
conda-package-handling    1.8.0            py37h7f8727e_0  
cryptography              36.0.0           py37h9ce1e76_0  
curl                      7.82.0               h7bff187_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.11.2           py37h5e8e339_1    conda-forge
dask-core                 1.1.4                    py37_1  
docutils                  0.18.1           py37h89c1867_0    conda-forge
enum34                    1.1.10           py37hc8dfbb8_2    conda-forge
et_xmlfile                1.0.1                   py_1001    conda-forge
expat                     2.4.7                h27087fc_0    conda-forge
fftw                      3.3.10          nompi_h77c792f_102    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
frozenlist                1.3.0            py37h5e8e339_0    conda-forge
fsspec                    2022.2.0           pyhd8ed1ab_0    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
git                       2.35.1          pl5321h36853c3_0    conda-forge
glpk                      4.65              h9202a9a_1004    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
h5py                      3.6.0           nompi_py37hd308b1e_100    conda-forge
hdf5                      1.12.1          nompi_h2386368_104    conda-forge
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      2.10               pyhd3eb1b0_0  
ilmbase                   2.5.5                h780b84a_0    conda-forge
imagecodecs-lite          2019.12.3        py37hb1e94ed_4    conda-forge
imageio                   2.6.1                    py37_0    conda-forge
importlib-metadata        4.11.3           py37h89c1867_0    conda-forge
importlib_resources       5.6.0              pyhd8ed1ab_0    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jmespath                  1.0.0              pyhd8ed1ab_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h7f98852_0    conda-forge
jsonschema                4.4.0              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.0            py37h7cecad7_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               h7274673_9  
lemon                     1.3.1                hf0e68b2_5    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           13_linux64_openblas    conda-forge
libcblas                  3.9.0           13_linux64_openblas    conda-forge
libcurl                   7.82.0               h7bff187_0    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.3                  he6710b0_2  
libgcc-ng                 11.2.0              h1d223b6_14    conda-forge
libgfortran-ng            11.2.0              h69a702a_14    conda-forge
libgfortran5              11.2.0              h5c6108e_14    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           13_linux64_openblas    conda-forge
liblapacke                3.9.0           13_linux64_openblas    conda-forge
libllvm11                 11.1.0               hf817b99_3    conda-forge
libnghttp2                1.47.0               h727a467_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.18          pthreads_h8fe5266_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_14    conda-forge
libtiff                   4.3.0                h542a066_3    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
llvm-openmp               13.0.1               he0ac6c6_1    conda-forge
llvmlite                  0.38.0           py37h9d7f4d0_0    conda-forge
lockfile                  0.12.2                     py_1    conda-forge
luigi                     3.0.3            py37h06a4308_0  
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
matplotlib-base           3.4.3            py37h1058ff1_0    conda-forge
mobie_utils               0.2.3              pyhd8ed1ab_0    conda-forge
mrcfile                   1.3.0              pyh44b312d_0    conda-forge
multidict                 6.0.2            py37h5e8e339_0    conda-forge
ncurses                   6.2                  he6710b0_1  
networkx                  2.7.1              pyhd8ed1ab_0    conda-forge
nifty                     1.2.1            py37h023b2db_3    conda-forge
numba                     0.55.1           py37h2d894fd_0    conda-forge
numpy                     1.21.5           py37hf2998dd_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openexr                   2.5.5                hf817b99_0    conda-forge
openpyxl                  3.0.9              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1n               h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py37he8f5f7f_0    conda-forge
pathlib                   1.0.1            py37h89c1867_5    conda-forge
pcre2                     10.37                h032f7d1_0    conda-forge
perl                      5.32.1          2_h7f98852_perl5    conda-forge
pillow                    7.2.0            py37h718be6c_2    conda-forge
pip                       21.2.2           py37h06a4308_0  
pybdv                     0.5.3              pyhd8ed1ab_0    conda-forge
pycosat                   0.6.3            py37h27cfd23_0  
pycparser                 2.20                       py_2  
pyopenssl                 20.0.1             pyhd3eb1b0_1  
pyparsing                 3.0.7              pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.1           py37h5e8e339_0    conda-forge
pysocks                   1.7.1                    py37_1  
python                    3.7.11               h12debd9_0  
python-daemon             2.3.0              pyhd8ed1ab_1    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-elf                0.4.1              pyhd8ed1ab_0    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pywavelets                1.3.0            py37h6c7ee08_0    conda-forge
readline                  8.1                  h27cfd23_0  
requests                  2.25.1             pyhd3eb1b0_0  
ruamel_yaml               0.15.100         py37h27cfd23_0  
s3fs                      0.5.1                      py_0    conda-forge
scikit-image              0.19.2           py37he8f5f7f_0    conda-forge
scikit-learn              1.0.2            py37hf9e9bfc_0    conda-forge
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
setuptools                58.0.4           py37h06a4308_0  
six                       1.16.0             pyhd3eb1b0_0  
skan                      0.10.0             pyhd8ed1ab_0    conda-forge
sqlite                    3.36.0               hc218d9a_0  
tenacity                  6.3.1              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tifffile                  2019.7.26.2              py37_0    conda-forge
tk                        8.6.10               hbc83047_0  
toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py37h5e8e339_2    conda-forge
tqdm                      4.61.2             pyhd3eb1b0_1  
typing-extensions         4.1.1                hd8ed1ab_0    conda-forge
typing_extensions         4.1.1              pyha770c72_0    conda-forge
tzdata                    2021a                h52ac0ba_0  
urllib3                   1.26.6             pyhd3eb1b0_1  
vigra                     1.11.1          py37h71a3c6e_1029    conda-forge
wheel                     0.36.2             pyhd3eb1b0_0  
wrapt                     1.14.0           py37h540881e_0    conda-forge
xz                        5.2.5                h7b6447c_0  
yaml                      0.2.5                h7b6447c_0  
yarl                      1.7.2            py37h5e8e339_1    conda-forge
z5py                      2.0.15           py37h2f61680_0    conda-forge
zipp                      3.7.0              pyhd8ed1ab_1    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge
constantinpape commented 2 years ago

The issue arises when running python as a non-privileged user inside the container.

By "the issue", do you mean the error reported in https://github.com/mobie/mobie-utils-python/issues/49#issuecomment-1081810124? If so, then this is probably caused by numba modifying files that are not accessible in the container. This is an issue I can't do much about; except for guarding against import errors of skan / numba in elf. But numba is a very commonly used package, so a CI set-up that can't deal with it is not very good.

martinschorb commented 2 years ago

Yes, it looks like it is the good old conda permissions issue in a multi-user environment.

If I chmod 777 all conda and install everything as the user, it works.

martinschorb commented 2 years ago

Cool, all the test container runs now.

Any idea why I get an old mobie_utils version when installing it from conda-forge?

       mobie.add_bdv_image(xmlpath,
                            outpath,
                            dataset_name=dataset_name,
                            image_name=image_name,
                            menu_name=menu_name,
>                           move_data=True)
E       TypeError: add_bdv_image() got an unexpected keyword argument 'move_data'

Do I still need to additionally pip install from the repo?

constantinpape commented 2 years ago

I haven't done a release yet since you've added the move_data option. I have done it now, but it will take probably till tomorrow to be on conda-forge. I will ping you then.

martinschorb commented 2 years ago

Cool, thanks.