Multiple installation issues + dependency resolution

peekxc commented 3 months ago

Part of 6773

I've encountered multiple installation issues trying to install from source, in part due to the sheer number of dependencies the project requires.

I'm on macOS, with my default compiler being clang 18.1.8 (homebrewed version), on x86_64.

Attempting to create a fresh python environment will all of the dependencies needed for multipers via mamba (or conda) currently doesn't work for my system

In particular,

mamba create -n multipers numpy scipy cxx-compiler boost tbb tbb-devel matplotlib gudhi scikit-learn cython sympy tqdm cycler typing shapely python==3.11

Yields multiple error messages of the form:

Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY ... error libmamba Could not solve for environment specs ... gudhi 3.8.0 would require │ └─ libcxx >=15.0.7 , which conflicts with any installable versions previously reported; critical libmamba Could not solve for environment specs

The first big issue seems to be conda version of gudhi, which introduces many outdated dependencies which can't resolve. For example, if I use the latest packages from all of those dependencies except gudhi (installed via pip), and then I run:

mamba install gudhi

Within said virtualenv, I recieve the following output:

dependency hell

```bash pkgs/main/noarch No change pkgs/r/osx-64 No change pkgs/r/noarch No change nodefaults/osx-64 No change nodefaults/noarch No change pkgs/main/osx-64 5.9MB @ 4.3MB/s 1.4s conda-forge/noarch 15.3MB @ 6.4MB/s 2.2s conda-forge/osx-64 31.6MB @ 6.0MB/s 5.3s Pinned packages: - python 3.11.* Transaction Prefix: /Users/mpiekenbrock/opt/miniconda3/envs/multipers Updating specs: - gudhi Package Version Build Channel Size ──────────────────────────────────────────────────────────────────────────────────────── Install: ──────────────────────────────────────────────────────────────────────────────────────── + pcre 8.45 h23ab428_0 pkgs/main Cached + pandoc 2.12 hecd8cb5_3 pkgs/main Cached + gmp 6.2.1 he9d5cce_3 pkgs/main Cached + expat 2.6.2 hcec6c5f_0 pkgs/main Cached + mpfr 4.0.2 h9066e36_1 pkgs/main Cached + ipython_genutils 0.2.0 pyhd3eb1b0_1 pkgs/main Cached + pyrsistent 0.17.3 py36haf1e3a3_0 pkgs/main Cached + entrypoints 0.3 py36_0 pkgs/main Cached + async_generator 1.10 pyhd3eb1b0_0 pkgs/main Cached + zipp 3.6.0 pyhd3eb1b0_0 pkgs/main Cached + dataclasses 0.8 pyh4f3eec9_6 pkgs/main Cached + testpath 0.5.0 pyhd3eb1b0_0 pkgs/main Cached + importlib-metadata 4.8.1 py36hecd8cb5_0 pkgs/main Cached + immutables 0.16 py36h9ed2024_0 pkgs/main Cached + importlib_metadata 4.8.1 hd3eb1b0_0 pkgs/main Cached + contextvars 2.4 py_0 pkgs/main Cached + cryptography 35.0.0 py36h2fd3fbb_0 pkgs/main Cached + brotlipy 0.7.0 py36h9ed2024_1003 pkgs/main Cached + dbus 1.13.18 h18a8e69_0 pkgs/main Cached + qt 5.9.7 h468cd18_1 pkgs/main Cached + boost-cpp 1.67.0 h1de35cc_4 pkgs/main Cached + backcall 0.2.0 pyhd3eb1b0_0 pkgs/main Cached + olefile 0.46 pyhd3eb1b0_0 pkgs/main Cached + pickleshare 0.7.5 pyhd3eb1b0_1003 pkgs/main Cached + munkres 1.1.4 py_0 pkgs/main 14kB + pyopenssl 22.0.0 pyhd3eb1b0_0 pkgs/main Cached + cgal 4.12 py36h8634a1c_1 conda-forge Cached + gudhi 2.3.0 py36h83d4a3d_0 conda-forge Cached + nbclassic 0.2.6 pyhd3eb1b0_0 pkgs/main Cached Remove: ──────────────────────────────────────────────────────────────────────────────────────── - contourpy 1.2.0 py311ha357a0b_0 pkgs/main Cached - libglib 2.78.4 h19e1a8f_0 pkgs/main Cached - pin-1 1 localhost - argon2-cffi-bindings 21.2.0 py311h6c40b1e_0 pkgs/main Cached - jupyter_events 0.10.0 py311hecd8cb5_0 pkgs/main Cached - referencing 0.30.2 py311hecd8cb5_0 pkgs/main Cached - rpds-py 0.10.6 py311hf2ad997_0 pkgs/main Cached - rfc3986-validator 0.1.1 py311hecd8cb5_0 pkgs/main Cached - comm 0.2.1 py311hecd8cb5_0 pkgs/main Cached - jupyter_server_terminals 0.4.4 py311hecd8cb5_1 pkgs/main Cached - tinycss2 1.2.1 py311hecd8cb5_0 pkgs/main Cached - jupyter-lsp 2.2.0 py311hecd8cb5_0 pkgs/main Cached - rfc3339-validator 0.1.4 py311hecd8cb5_0 pkgs/main Cached - qt-main 5.15.2 h17e5239_10 pkgs/main Cached Change: ──────────────────────────────────────────────────────────────────────────────────────── - libllvm14 14.0.6 h91fad77_3 pkgs/main Cached + libllvm14 14.0.6 he552d86_0 pkgs/main Cached - llvm-tools 14.0.6 he0576d7_3 pkgs/main Cached + llvm-tools 14.0.6 h8bd2559_0 pkgs/main Cached - webencodings 0.5.1 py311hecd8cb5_1 pkgs/main Cached + webencodings 0.5.1 py36_1 pkgs/main Cached - pysocks 1.7.1 py311hecd8cb5_0 pkgs/main Cached + pysocks 1.7.1 py36hecd8cb5_0 pkgs/main Cached - appnope 0.1.2 py311hecd8cb5_1001 pkgs/main Cached + appnope 0.1.2 py36hecd8cb5_1001 pkgs/main Cached - ply 3.11 py311hecd8cb5_0 pkgs/main Cached + ply 3.11 py36_0 pkgs/main Cached - brotli-python 1.0.9 py311hcec6c5f_8 pkgs/main Cached + brotli-python 1.0.9 py36h79c402e_2 pkgs/main Cached - libxml2 2.10.4 h45904e2_2 pkgs/main Cached + libxml2 2.10.4 h930c0e2_0 pkgs/main Cached - jupyter 1.0.0 py311hecd8cb5_9 pkgs/main Cached + jupyter 1.0.0 py36_7 pkgs/main Cached Reinstall: ──────────────────────────────────────────────────────────────────────────────────────── o pycparser 2.21 pyhd3eb1b0_0 pkgs/main Cached o six 1.16.0 pyhd3eb1b0_1 pkgs/main Cached o decorator 5.1.1 pyhd3eb1b0_0 pkgs/main Cached o ptyprocess 0.7.0 pyhd3eb1b0_2 pkgs/main Cached o pandocfilters 1.5.0 pyhd3eb1b0_0 pkgs/main Cached o defusedxml 0.7.1 pyhd3eb1b0_0 pkgs/main Cached o jupyterlab_pygments 0.1.2 py_0 pkgs/main Cached o bleach 4.1.0 pyhd3eb1b0_0 pkgs/main Cached o wcwidth 0.2.5 pyhd3eb1b0_0 pkgs/main Cached o python-tzdata 2023.3 pyhd3eb1b0_0 pkgs/main Cached o pure_eval 0.2.2 pyhd3eb1b0_0 pkgs/main Cached o pexpect 4.8.0 pyhd3eb1b0_3 pkgs/main Cached o parso 0.8.3 pyhd3eb1b0_0 pkgs/main Cached o json5 0.9.6 pyhd3eb1b0_0 pkgs/main Cached o executing 0.8.3 pyhd3eb1b0_0 pkgs/main Cached o cycler 0.11.0 pyhd3eb1b0_0 pkgs/main Cached o charset-normalizer 2.0.4 pyhd3eb1b0_0 pkgs/main Cached o asttokens 2.0.5 pyhd3eb1b0_0 pkgs/main Cached o stack_data 0.2.0 pyhd3eb1b0_0 pkgs/main Cached Downgrade: ──────────────────────────────────────────────────────────────────────────────────────── - libffi 3.4.4 hecd8cb5_1 pkgs/main Cached + libffi 3.3 hb1e8313_2 pkgs/main Cached - python 3.11.0 h1fd4e5f_3 pkgs/main Cached + python 3.6.13 h88f2d9e_0 pkgs/main Cached - certifi 2024.6.2 py311hecd8cb5_0 pkgs/main Cached + certifi 2021.5.30 py36hecd8cb5_0 pkgs/main Cached - setuptools 69.5.1 py311hecd8cb5_0 pkgs/main Cached + setuptools 58.0.4 py36hecd8cb5_0 pkgs/main Cached - wheel 0.43.0 py311hecd8cb5_0 pkgs/main Cached + wheel 0.37.1 pyhd3eb1b0_0 pkgs/main Cached - pip 24.0 py311hecd8cb5_0 pkgs/main Cached + pip 21.2.2 py36hecd8cb5_0 pkgs/main Cached - cffi 1.16.0 py311h6c40b1e_1 pkgs/main Cached + cffi 1.14.6 py36h2125817_0 pkgs/main Cached - pyyaml 6.0.1 py311h6c40b1e_0 pkgs/main Cached + pyyaml 5.4.1 py36h9ed2024_1 pkgs/main Cached - traitlets 5.14.3 py311hecd8cb5_0 pkgs/main Cached + traitlets 4.3.3 py36hecd8cb5_0 pkgs/main Cached - tornado 6.4.1 py311h46256e1_0 pkgs/main Cached + tornado 6.1 py36h9ed2024_0 pkgs/main Cached - terminado 0.17.1 py311hecd8cb5_0 pkgs/main Cached + terminado 0.9.4 py36hecd8cb5_0 pkgs/main Cached - pyzmq 25.1.2 py311hcec6c5f_0 pkgs/main Cached + pyzmq 22.2.1 py36h23ab428_1 pkgs/main Cached - jupyter_core 5.7.2 py311hecd8cb5_0 pkgs/main Cached + jupyter_core 4.8.1 py36hecd8cb5_0 pkgs/main Cached - mistune 2.0.4 py311hecd8cb5_0 pkgs/main Cached + mistune 0.8.4 py36h1de35cc_0 pkgs/main Cached - markupsafe 2.1.3 py311h6c40b1e_0 pkgs/main Cached + markupsafe 2.0.1 py36h9ed2024_0 pkgs/main Cached - typing_extensions 4.11.0 py311hecd8cb5_0 pkgs/main Cached + typing_extensions 4.1.1 pyh06a4308_0 pkgs/main Cached - soupsieve 2.5 py311hecd8cb5_0 pkgs/main Cached + soupsieve 2.3.1 pyhd3eb1b0_0 pkgs/main Cached - send2trash 1.8.2 py311hecd8cb5_0 pkgs/main Cached + send2trash 1.8.0 pyhd3eb1b0_1 pkgs/main Cached - python-fastjsonschema 2.16.2 py311hecd8cb5_0 pkgs/main Cached + python-fastjsonschema 2.15.1 pyhd3eb1b0_0 pkgs/main Cached - python-dateutil 2.9.0post0 py311hecd8cb5_2 pkgs/main Cached + python-dateutil 2.8.2 pyhd3eb1b0_0 pkgs/main Cached - idna 3.7 py311hecd8cb5_0 pkgs/main Cached + idna 3.3 pyhd3eb1b0_0 pkgs/main Cached - attrs 23.1.0 py311hecd8cb5_0 pkgs/main Cached + attrs 21.4.0 pyhd3eb1b0_0 pkgs/main Cached - pyparsing 3.0.9 py311hecd8cb5_0 pkgs/main Cached + pyparsing 3.0.4 pyhd3eb1b0_0 pkgs/main Cached - pygments 2.15.1 py311hecd8cb5_1 pkgs/main Cached + pygments 2.11.2 pyhd3eb1b0_0 pkgs/main Cached - prometheus_client 0.14.1 py311hecd8cb5_0 pkgs/main Cached + prometheus_client 0.13.1 pyhd3eb1b0_0 pkgs/main Cached - nest-asyncio 1.6.0 py311hecd8cb5_0 pkgs/main Cached + nest-asyncio 1.5.1 pyhd3eb1b0_0 pkgs/main Cached - jinja2 3.1.4 py311hecd8cb5_0 pkgs/main Cached + jinja2 3.0.3 pyhd3eb1b0_0 pkgs/main Cached - typing-extensions 4.11.0 py311hecd8cb5_0 pkgs/main Cached + typing-extensions 4.1.1 hd3eb1b0_0 pkgs/main Cached - beautifulsoup4 4.12.3 py311hecd8cb5_0 pkgs/main Cached + beautifulsoup4 4.10.0 pyh06a4308_0 pkgs/main Cached - packaging 23.2 py311hecd8cb5_0 pkgs/main Cached + packaging 21.3 pyhd3eb1b0_0 pkgs/main Cached - jupyter_client 8.6.0 py311hecd8cb5_0 pkgs/main Cached + jupyter_client 7.1.2 pyhd3eb1b0_0 pkgs/main Cached - jsonschema 4.19.2 py311hecd8cb5_0 pkgs/main Cached + jsonschema 3.2.0 pyhd3eb1b0_2 pkgs/main Cached - nbformat 5.9.2 py311hecd8cb5_0 pkgs/main Cached + nbformat 5.1.3 pyhd3eb1b0_0 pkgs/main Cached - nbclient 0.8.0 py311hecd8cb5_0 pkgs/main Cached + nbclient 0.5.3 pyhd3eb1b0_0 pkgs/main Cached - sniffio 1.3.0 py311hecd8cb5_0 pkgs/main Cached + sniffio 1.2.0 py36hecd8cb5_1 pkgs/main Cached - nbconvert 7.10.0 py311hecd8cb5_0 pkgs/main Cached + nbconvert 6.0.7 py36_0 pkgs/main Cached - anyio 4.2.0 py311hecd8cb5_0 pkgs/main Cached + anyio 2.2.0 py36hecd8cb5_1 pkgs/main Cached - jupyter_server 2.14.1 py311hecd8cb5_0 pkgs/main Cached + jupyter_server 1.4.1 py36hecd8cb5_0 pkgs/main Cached - websocket-client 1.8.0 py311hecd8cb5_0 pkgs/main Cached + websocket-client 0.58.0 py36hecd8cb5_4 pkgs/main Cached - unicodedata2 15.1.0 py311h6c40b1e_0 pkgs/main Cached + unicodedata2 13.0.0.post2 py36h9ed2024_0 pkgs/main 380kB - kiwisolver 1.4.4 py311hcec6c5f_0 pkgs/main Cached + kiwisolver 1.3.1 py36h23ab428_0 pkgs/main Cached - glib 2.78.4 hcec6c5f_0 pkgs/main Cached + glib 2.69.1 h8346a28_1 pkgs/main Cached - debugpy 1.6.7 py311hcec6c5f_0 pkgs/main Cached + debugpy 1.4.1 py36h23ab428_0 pkgs/main Cached - argon2-cffi 21.3.0 pyhd3eb1b0_0 pkgs/main Cached + argon2-cffi 20.1.0 py36h9ed2024_1 pkgs/main Cached - psutil 5.9.0 py311h6c40b1e_0 pkgs/main Cached + psutil 5.8.0 py36h9ed2024_1 pkgs/main Cached - numpy-base 1.26.4 py311hb3ec012_0 pkgs/main Cached + numpy-base 1.19.2 py36h68fea81_0 pkgs/main Cached - icu 73.1 hcec6c5f_0 pkgs/main Cached + icu 58.2 h0a44026_3 pkgs/main Cached - sip 6.7.12 py311hcec6c5f_0 pkgs/main Cached + sip 4.19.8 py36h0a44026_0 pkgs/main Cached - numpy 1.26.4 py311h91b6869_0 pkgs/main Cached + numpy 1.19.2 py36h63973fd_0 pkgs/main Cached - libboost 1.82.0 hf53b9f2_2 pkgs/main Cached + libboost 1.67.0 hebc422b_4 pkgs/main Cached - bottleneck 1.3.7 py311hb3a5e46_0 pkgs/main Cached + bottleneck 1.3.2 py36hf1fa96c_1 pkgs/main Cached - numexpr 2.8.7 py311h91b6869_0 pkgs/main Cached + numexpr 2.7.3 py36h7ec9b2a_1 pkgs/main Cached - pyqt 5.15.10 py311hcec6c5f_0 pkgs/main Cached + pyqt 5.9.2 py36h655552a_2 pkgs/main Cached - py-boost 1.82.0 py311h7966749_2 pkgs/main Cached + py-boost 1.67.0 py36h6440ff4_4 pkgs/main Cached - boost 1.82.0 py311hecd8cb5_2 pkgs/main Cached + boost 1.67.0 py36_4 pkgs/main Cached - pytz 2024.1 py311hecd8cb5_0 pkgs/main Cached + pytz 2021.3 pyhd3eb1b0_0 pkgs/main Cached - python-json-logger 2.0.7 py311hecd8cb5_0 pkgs/main Cached + python-json-logger 2.0.1 py_0 pkgs/main Cached - jupyterlab_widgets 3.0.10 py311hecd8cb5_0 pkgs/main Cached + jupyterlab_widgets 1.0.0 pyhd3eb1b0_1 pkgs/main Cached - qtpy 2.4.1 py311hecd8cb5_0 pkgs/main Cached + qtpy 2.0.1 pyhd3eb1b0_0 pkgs/main Cached - platformdirs 3.10.0 py311hecd8cb5_0 pkgs/main Cached + platformdirs 2.4.0 pyhd3eb1b0_0 pkgs/main Cached - matplotlib-inline 0.1.6 py311hecd8cb5_0 pkgs/main Cached + matplotlib-inline 0.1.2 pyhd3eb1b0_2 pkgs/main Cached - prompt-toolkit 3.0.43 py311hecd8cb5_0 pkgs/main Cached + prompt-toolkit 3.0.20 pyhd3eb1b0_0 pkgs/main Cached - fonttools 4.51.0 py311h6c40b1e_0 pkgs/main Cached + fonttools 4.25.0 pyhd3eb1b0_0 pkgs/main 648kB - babel 2.11.0 py311hecd8cb5_0 pkgs/main Cached + babel 2.9.1 pyhd3eb1b0_0 pkgs/main Cached - urllib3 2.2.2 py311hecd8cb5_0 pkgs/main Cached + urllib3 1.26.8 pyhd3eb1b0_0 pkgs/main Cached - prompt_toolkit 3.0.43 hd3eb1b0_0 pkgs/main Cached + prompt_toolkit 3.0.20 hd3eb1b0_0 pkgs/main Cached - requests 2.32.2 py311hecd8cb5_0 pkgs/main Cached + requests 2.27.1 pyhd3eb1b0_0 pkgs/main Cached - jupyterlab_server 2.25.1 py311hecd8cb5_0 pkgs/main Cached + jupyterlab_server 2.10.3 pyhd3eb1b0_1 pkgs/main Cached - jedi 0.18.1 py311hecd8cb5_1 pkgs/main Cached + jedi 0.17.0 py36_0 pkgs/main Cached - pillow 10.3.0 py311h6c40b1e_0 pkgs/main Cached + pillow 8.3.1 py36ha4cf6ea_0 pkgs/main Cached - pandas 2.2.2 py311he327ffe_0 pkgs/main Cached + pandas 1.1.5 py36hb2f4e1b_0 pkgs/main Cached - ipython 8.25.0 py311hecd8cb5_0 pkgs/main Cached + ipython 7.16.1 py36h5ca1d4c_0 pkgs/main Cached - bokeh 3.4.1 py311h85bffb1_0 pkgs/main Cached + bokeh 2.3.2 py36hecd8cb5_0 pkgs/main Cached - matplotlib-base 3.8.4 py311h41a4f6b_0 pkgs/main Cached + matplotlib-base 3.3.4 py36h8b3ea08_0 pkgs/main Cached - ipykernel 6.28.0 py311hecd8cb5_0 pkgs/main Cached + ipykernel 5.3.4 py36h5ca1d4c_0 pkgs/main Cached - matplotlib 3.8.4 py311hecd8cb5_0 pkgs/main Cached + matplotlib 3.3.4 py36hecd8cb5_0 pkgs/main Cached - notebook 7.0.8 py311hecd8cb5_0 pkgs/main Cached + notebook 6.4.3 py36hecd8cb5_0 pkgs/main Cached - widgetsnbextension 4.0.10 py311hecd8cb5_0 pkgs/main Cached + widgetsnbextension 3.5.1 py36_0 pkgs/main Cached - jupyter_console 6.6.3 py311hecd8cb5_0 pkgs/main Cached + jupyter_console 6.4.3 pyhd3eb1b0_0 pkgs/main Cached - qtconsole 5.5.1 py311hecd8cb5_0 pkgs/main Cached + qtconsole 5.2.2 pyhd3eb1b0_0 pkgs/main Cached - ipywidgets 8.1.2 py311hecd8cb5_0 pkgs/main Cached + ipywidgets 7.6.5 pyhd3eb1b0_2 pkgs/main Cached - jupyterlab 4.0.11 py311hecd8cb5_0 pkgs/main Cached + jupyterlab 3.2.1 pyhd3eb1b0_1 pkgs/main Cached Summary: Install: 29 packages Remove: 14 packages Change: 9 packages Reinstall: 19 packages Downgrade: 82 packages Total download: 1MB ──────────────────────────────────────────────────────────────────────────────────────── ```

Then, I actually run into issues even in the install, recieving an unknown error of the following:

Transaction starting
munkres                                             13.6kB @ 132.1kB/s  0.1s
unicodedata2                                       380.2kB @   2.2MB/s  0.2s
fonttools                                          647.6kB @   3.6MB/s  0.2s
Unlinking contourpy-1.2.0-py311ha357a0b_0
Unlinking libglib-2.78.4-h19e1a8f_0
Unlinking pin-1-1-
critical libmamba Cannot strip file extension from:

If I just install gudhi from pip, the wheel installs fine.

The second big issue for me comes from trying to compile from source. I recieve hundreds of errors, though they all seem to be of the same type (and linked to TBB), they're all like:

  /Users/mpiekenbrock/opt/miniconda3/envs/multipers/include/tbb/../oneapi/tbb/detail/_task.h:216:13: error: aligned deallocation function of type 'void (void *, std::align_val_t) noexcept' is only available on macOS 10.14 or newer
      virtual ~task() = default;
              ^

Attached below is the full log, though it's practically the same thing repeated many times.

multipers_cpp.txt

Interesting, I can actually install gudhi-devel perfectly fine; I think that's likley due to the fact that they use cmake (which uses pkgconfig) to find these large system libraries I've already installed in my system long ago (like tbb), whereas it looks like multipers is relying on a pure python-dependency approach, which might not support linking as smoothly as pkgconfig

peekxc commented 3 months ago

(I should add that installing multipers from pypi via pip works fine on my machine, this is just about installing from source)

DavidLapous commented 3 months ago

Thanks for this investigation.

For the dependencies, I need to investigate a bit, as, in the case of "from source compilation", I haven't updated that in a while; My issue is that I want to use the less source as possible (i.e., only from conda) and have similar installation between OSes which may add unnecessary problems. Also, I don't have access to a macOS machine, so if you have a working environment, I'd be happy to have it.

According to the current GitHub workflow (which builds + tests + sends to pypi) the following environment should work without any issue on any up-to-date Linux or macOS machine

mamba install boost-cpp  gudhi numpy=2 cython pytest scikit-learn matplotlib boost joblib tqdm scipy tbb tbb-devel -c conda-forge
pip install pykeops filtration-domination pot --upgrade

Regarding your compilation error, this is due to an (old) Apple clang issue, which I used to point out, but I forgot to re-add to the documentation when I rewrote it. This should be solved by disabling a compilation optimization, by uncommenting a line in the setup file. There is also this workaround in the workflow which is possible.

peekxc commented 1 month ago

I've been able to resolve my installation issues. In particular, using a fresh Python 3.10 (or 3.11) environment + installing multipers the recommended way looks to now be working.

DavidLapous / multipers

Multiple installation issues + dependency resolution #12