Broken pyzmq installation due to libsodium not being updated alongside

megies commented 5 years ago

Issue:

It seems that pyzmq packages depend on specific versions of libsodium, but the package metadata does not specify them. Thus e.g. by updating pyzmq, conda does not also update (or rather install the exact needed version of) libsodium at the same time, which leaves the updated pyzmq broken.

I think this should be resolved in one of the following ways:

if pyzmq does not need a specific libsodium version (implied by no dep version specified in recipe): pyzmq should look up libsodium.so which will link to the specific version currently installed in the system
if pyzmq does need a specific libsodium version: package metadata should state that version of libsodium so conda can resolve the dependency correctly

Steps to reproduce

Step 1: outdated pyzmq version installed

Conda env with..

pyzmq 16.0.2-py27_2
libsodium 1.0.10

Step 2: Update `pyzmq`

Update pyzmq using conda: conda install -c conda-forge pyzmq

``` $ conda install -c conda-forge pyzmq Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.5.4 latest version: 4.5.11 Please update conda by running $ conda update -n base conda ## Package Plan ## environment location: /home/megies/anaconda/envs/default added / updated specs: - pyzmq The following packages will be downloaded: package | build ---------------------------|----------------- zeromq-4.2.5 | hfc679d8_4 939 KB conda-forge pyzmq-17.1.2 | py27hae99301_0 969 KB conda-forge ------------------------------------------------------------ Total: 1.9 MB The following packages will be UPDATED: certifi: 2018.8.24-py27_1 conda-forge --> 2018.8.24-py27_1001 conda-forge openssl: 1.0.2o-h470a237_1 conda-forge --> 1.0.2p-h470a237_0 conda-forge pyzmq: 16.0.2-py27_2 conda-forge --> 17.1.2-py27hae99301_0 conda-forge zeromq: 4.2.1-1 conda-forge --> 4.2.5-hfc679d8_4 conda-forge Proceed ([y]/n)? Downloading and Extracting Packages zeromq-4.2.5 | 939 KB | ######################################################################################### | 100% pyzmq-17.1.2 | 969 KB | ######################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done ```

Step 3: Broken `pyzmq` installation

Broken pyzmq due to mismatching libsodium version. pyzmq looks specifically for libsodium.so.23 but instead in the system libsodium.so.18 is installed due to version mismatch.

$ ll /home/megies/anaconda/envs/default/lib/libsodium.*
-rw-r--r-- 5 megies users 4125302 Jun 24  2016 /home/megies/anaconda/envs/default/lib/libsodium.a
-rwxr-xr-x 1 megies users     958 May 11  2017 /home/megies/anaconda/envs/default/lib/libsodium.la*
lrwxrwxrwx 1 megies users      19 May 11  2017 /home/megies/anaconda/envs/default/lib/libsodium.so -> libsodium.so.18.1.0*
lrwxrwxrwx 1 megies users      19 May 11  2017 /home/megies/anaconda/envs/default/lib/libsodium.so.18 -> libsodium.so.18.1.0*
-rwxr-xr-x 5 megies users 2409968 Jun 24  2016 /home/megies/anaconda/envs/default/lib/libsodium.so.18.1.0*

``` $ jupyter notebook --no-browser Traceback (most recent call last): File "/home/megies/anaconda/envs/default/bin/jupyter-notebook", line 4, in import notebook.notebookapp File "/home/megies/anaconda/envs/default/lib/python2.7/site-packages/notebook/notebookapp.py", line 39, in from zmq.eventloop import ioloop File "/home/megies/anaconda/envs/default/lib/python2.7/site-packages/zmq/__init__.py", line 47, in from zmq import backend File "/home/megies/anaconda/envs/default/lib/python2.7/site-packages/zmq/backend/__init__.py", line 40, in reraise(*exc_info) File "/home/megies/anaconda/envs/default/lib/python2.7/site-packages/zmq/backend/__init__.py", line 27, in _ns = select_backend(first) File "/home/megies/anaconda/envs/default/lib/python2.7/site-packages/zmq/backend/select.py", line 27, in select_backend mod = __import__(name, fromlist=public_api) File "/home/megies/anaconda/envs/default/lib/python2.7/site-packages/zmq/backend/cython/__init__.py", line 6, in from . import (constants, error, message, context, ImportError: libsodium.so.23: cannot open shared object file: No such file or directory ```

Step 4: Fix env by manually updating `libsodium`

Updating libsodium makes everything work again.

``` $ conda install -c conda-forge libsodium Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.5.4 latest version: 4.5.11 Please update conda by running $ conda update -n base conda ## Package Plan ## environment location: /home/megies/anaconda/envs/default added / updated specs: - libsodium The following packages will be downloaded: package | build ---------------------------|----------------- libsodium-1.0.16 | h470a237_1 536 KB conda-forge The following packages will be UPDATED: libsodium: 1.0.10-0 conda-forge --> 1.0.16-h470a237_1 conda-forge Proceed ([y]/n)? Downloading and Extracting Packages libsodium-1.0.16 | 536 KB | ######################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done $ jupyter notebook --no-browser [I 12:37:54.970 NotebookApp] Writing notebook server cookie secret to /run/user/1043/jupyter/notebook_cookie_secret [W 12:37:55.324 NotebookApp] Error loading server extension jupyterlab Traceback (most recent call last): File "/home/megies/anaconda/envs/default/lib/python2.7/site-packages/notebook/notebookapp.py", line 1271, in init_server_extensions mod = importlib.import_module(modulename) File "/home/megies/anaconda/envs/default/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) ImportError: No module named jupyterlab [I 12:37:55.332 NotebookApp] Serving notebooks from local directory: /home/megies/notebooks [I 12:37:55.332 NotebookApp] 0 active kernels [I 12:37:55.332 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=e5fc8d435f602183f20846ba57fbff1b61cb99203b5ddcf7 [I 12:37:55.332 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 12:37:55.333 NotebookApp] [...] ```

Environment (`conda list`):

``` $ conda list # packages in environment at /home/megies/anaconda/envs/default: # # Name Version Build Channel alabaster 0.7.10 py27_1 conda-forge attrs 17.4.0 py_0 conda-forge babel 2.5.1 py27_0 conda-forge backports 1.0 py27_1 conda-forge backports.functools-lru-cache 1.3 backports.functools_lru_cache 1.4 py27_1 conda-forge backports.shutil_get_terminal_size 1.0.0 py27_1 conda-forge backports_abc 0.5 py27_0 conda-forge basemap 1.1.0 py27_3 conda-forge basemap-data-hires 1.1.0 0 conda-forge blas 1.0 mkl bleach 1.5.0 py27_0 conda-forge boost 1.64.0 py27_4 conda-forge boost-cpp 1.64.0 1 conda-forge bzip2 1.0.6 1 conda-forge ca-certificates 2018.8.24 ha4d7672_0 conda-forge cairo 1.14.10 0 conda-forge cartopy 0.16.0 py27h81b52dc_1 conda-forge certifi 2018.8.24 py27_1001 conda-forge click 6.7 py27_0 conda-forge click-plugins 1.0.3 py27_0 conda-forge cligj 0.4.0 py27_0 conda-forge configparser 3.5.0 py27_0 conda-forge curl 7.55.1 0 conda-forge cycler 0.10.0 py27_0 conda-forge dbus 1.10.10 3 conda-forge decorator 4.0.11 py27_0 conda-forge descartes 1.1.0 py27_0 conda-forge docker 2.5.1 docker-pycreds 0.2.1 py27_0 conda-forge docutils 0.14 py27_0 conda-forge entrypoints 0.2.2 py27_1 conda-forge enum34 1.1.6 py27_1 conda-forge execnet 1.3.0 py27_1 expat 2.2.5 0 conda-forge fancycompleter 0.7 py27_0 conda-forge fiona 1.7.9 py27_0 conda-forge flake8 3.3.0 py27_0 conda-forge fontconfig 2.12.6 0 conda-forge freetype 2.8.1 0 conda-forge freexl 1.0.2 1 conda-forge funcsigs 1.0.2 py27_0 conda-forge functools32 3.2.3.2 py27_1 conda-forge future 0.16.0 py27_0 conda-forge futures 3.2.0 py27_0 conda-forge gdal 2.0.0 py27_1 geographiclib 1.48 py27_0 conda-forge geojson 2.1.0 py_0 conda-forge geopandas 0.3.0 py27_0 conda-forge geopy 1.11.0 py27_0 conda-forge geos 3.6.2 1 conda-forge geotiff 1.4.2 1 conda-forge gettext 0.19.7 1 conda-forge giflib 5.1.4 0 conda-forge glib 2.55.0 0 conda-forge gmp 6.1.2 0 conda-forge graphite2 1.3.9 0 conda-forge greenlet 0.4.12 py27_0 conda-forge gst-plugins-base 1.8.0 0 conda-forge gstreamer 1.8.0 1 conda-forge harfbuzz 1.7.6 0 conda-forge hdf4 4.2.12 0 conda-forge hdf5 1.10.1 1 conda-forge html5lib 0.999 py27_0 conda-forge icu 58.2 0 conda-forge imagesize 0.7.1 py27_0 conda-forge intel-openmp 2018.0.3 0 ipaddress 1.0.18 py27_0 conda-forge ipykernel 4.6.1 py27_0 conda-forge ipython 5.4.1 py27_0 conda-forge ipython_genutils 0.2.0 py27_0 conda-forge ipywidgets 6.0.0 py27_0 conda-forge jinja2 2.9.5 py27_0 conda-forge jpeg 9b 0 conda-forge json-c 0.12 0 conda-forge jsonschema 2.5.1 py27_0 conda-forge jupyter 1.0.0 py27_0 conda-forge jupyter_client 5.0.1 py27_0 conda-forge jupyter_console 5.1.0 py27_0 conda-forge jupyter_core 4.3.0 py27_0 conda-forge kealib 1.4.7 4 conda-forge kiwisolver 1.0.1 py27_1 conda-forge krb5 1.14.2 0 conda-forge libboost 1.65.1 habcd387_4 libconda 4.0.0 py27_0 conda-forge libdap4 3.18.3 2 conda-forge libffi 3.2.1 3 conda-forge libgcc-ng 7.2.0 hdf63c60_3 libgdal 2.1.4 9 conda-forge libgfortran 3.0.0 1 libgfortran-ng 7.2.0 hdf63c60_3 libiconv 1.15 0 conda-forge libkml 1.3.0 h9d32c78_3 libnetcdf 4.4.1.1 10 conda-forge libpng 1.6.34 0 conda-forge libpq 9.5.4 4 conda-forge libsodium 1.0.16 h470a237_1 conda-forge libspatialindex 1.8.5 1 conda-forge libspatialite 4.3.0a 19 conda-forge libssh2 1.8.0 2 conda-forge libstdcxx-ng 7.2.0 hdf63c60_3 libtiff 4.0.9 0 conda-forge libxcb 1.12 1 conda-forge libxml2 2.9.4 4 conda-forge libxslt 1.1.29 3 conda-forge livereload 2.5.2 py_0 conda-forge lxml 3.7.3 py27_0 conda-forge markdown 2.6.11 py_0 conda-forge markupsafe 0.23 py27_1 conda-forge matplotlib 2.2.2 py27_0 conda-forge mccabe 0.6.1 py27_0 conda-forge mistune 0.7.4 py27_0 conda-forge mkdocs 1.0.2 py_1 conda-forge mkdocs-bootstrap 1.0 py_0 conda-forge mkdocs-bootswatch 1.0 py_0 conda-forge mkl 2018.0.3 1 mkl_fft 1.0.2 py27_0 conda-forge mkl_random 1.0.1 py27_0 conda-forge mock 2.0.0 py27_0 conda-forge mplstereonet 0.6.dev0 msgpack-python 0.4.8 py27_0 conda-forge munch 2.2.0 py27_0 conda-forge nbconvert 5.3.1 py_1 conda-forge nbformat 4.4.0 py27_0 conda-forge nbstripout 0.3.1 py27_0 conda-forge ncurses 5.9 10 conda-forge neovim 0.2.0 py27_0 conda-forge nllgrid 0.1 notebook 5.0.0 py27_0 conda-forge notedown 1.5.0 numpy 1.14.5 py27hcd700cb_0 numpy-base 1.14.5 py27hdbf6ddf_0 obspy 1.1.0 py27_1 conda-forge obspy 1.1.1rc1.post0+42.ge594292a3d.obspy.maintenance.1.1.x olefile 0.44 py27_0 conda-forge openblas 0.2.20 5 conda-forge openjpeg 2.3.0 2 conda-forge openssl 1.0.2p h470a237_0 conda-forge owslib 0.16.0 py_0 conda-forge pandas 0.20.3 py27_1 conda-forge pandoc 1.19.2 0 conda-forge pandoc-attributes 0.1.7 py27_0 conda-forge pandocfilters 1.4.1 py27_0 conda-forge pango 1.40.14 0 conda-forge pathlib2 2.2.1 py27_0 conda-forge patsy 0.4.1 py27_0 conda-forge pbr 3.0.0 py27_0 conda-forge pcre 8.39 0 conda-forge pdbpp 0.9.1 py27_0 conda-forge pep8-naming 0.4.1 py27_0 conda-forge pexpect 4.2.1 py27_0 conda-forge pickleshare 0.7.3 py27_0 conda-forge pillow 5.0.0 py27_0 conda-forge pip 9.0.1 py27_0 conda-forge pixman 0.34.0 0 conda-forge pkginfo 1.4.1 py27_0 conda-forge pluggy 0.6.0 py_0 conda-forge poppler 0.63.0 0 conda-forge poppler-data 0.4.7 0 conda-forge proj4 4.9.3 3 conda-forge prompt_toolkit 1.0.14 py27_0 conda-forge psutil 5.2.1 py27_0 conda-forge psycopg2 2.7.3.1 py27_0 conda-forge ptyprocess 0.5.1 py27_0 conda-forge py 1.5.2 py_0 conda-forge pycodestyle 2.3.1 py27_0 conda-forge pycosat 0.6.1 py27_0 conda-forge pyepsg 0.3.2 py_1 conda-forge pyflakes 1.5.0 py27_0 conda-forge pygments 2.2.0 py27_0 conda-forge pyparsing 2.2.0 py27_0 conda-forge pyproj 1.9.5.1 py27_0 conda-forge pyqt 5.6.0 py27_2 conda-forge pyrepl 0.8.4 py27_1 conda-forge pysal 1.14.2 py_0 conda-forge pyshp 1.2.11 py27_0 conda-forge pytest 3.4.2 py27_0 conda-forge pytest-xdist 1.16.0 py27_0 conda-forge python 2.7.14 4 conda-forge python-dateutil 2.6.0 py27_0 conda-forge pytz 2017.2 py27_0 conda-forge pyyaml 3.12 py27_1 conda-forge pyzmq 17.1.2 py27hae99301_0 conda-forge qt 5.6.2 7 conda-forge qtconsole 4.3.0 py27_0 conda-forge readline 7.0 0 conda-forge requests 2.13.0 py27_0 conda-forge requests-toolbelt 0.8.0 py27_0 conda-forge rtree 0.8.3 py27_0 conda-forge scandir 1.5 py27_1 conda-forge scipy 1.1.0 py27hfc37229_0 scons 2.5.1 py27_0 conda-forge seaborn 0.8.1 py27_0 conda-forge seedlink-plotter 0.1.0 setuptools 38.4.0 py27_0 conda-forge shapely 1.6.4 py27_0 conda-forge simplegeneric 0.8.1 py27_0 conda-forge singledispatch 3.4.0.3 py27_0 conda-forge sip 4.18 py27_1 conda-forge six 1.10.0 py27_1 conda-forge snowballstemmer 1.2.1 py27_0 conda-forge sphinx 1.6.7 py27_0 conda-forge sphinx_rtd_theme 0.2.4 py27_0 conda-forge sphinxcontrib-websupport 1.0.1 py27_0 conda-forge sqlalchemy 1.1.5 py27_0 conda-forge sqlite 3.20.1 2 conda-forge ssl_match_hostname 3.5.0.1 py27_1 conda-forge statsmodels 0.8.0 py27_0 conda-forge subprocess32 3.2.7 py27_0 conda-forge terminado 0.6 py27_0 conda-forge testpath 0.3 py27_0 conda-forge tk 8.6.7 0 conda-forge tornado 5.1 py27h470a237_1 conda-forge tqdm 4.19.4 py_0 conda-forge traitlets 4.3.2 py27_0 conda-forge trollius 2.1 py27_0 conda-forge twine 1.9.1 py27_2 conda-forge typing 3.6.2 py27_0 conda-forge util-linux 2.21 0 utm 0.4.2 py27_0 conda-forge wcwidth 0.1.7 py27_0 conda-forge webencodings 0.5 py27_0 conda-forge websocket-client 0.44.0 py27_0 conda-forge wheel 0.29.0 py27_0 conda-forge widgetsnbextension 2.0.0 py27_0 conda-forge wmctrl 0.3 py27_0 conda-forge xerces-c 3.2.0 0 conda-forge xorg-libxau 1.0.8 3 conda-forge xorg-libxdmcp 1.1.2 3 conda-forge xz 5.2.2 0 conda-forge yaml 0.1.6 0 conda-forge zeromq 4.2.5 hfc679d8_4 conda-forge zlib 1.2.11 0 conda-forge ```

Details about `conda` and system ( `conda info` ):

``` $ conda info active environment : default active env location : /home/megies/anaconda/envs/default shell level : 2 user config file : /home/megies/.condarc populated config files : /home/megies/.condarc conda version : 4.5.4 conda-build version : 3.7.1 python version : 2.7.11.final.0 base environment : /home/megies/anaconda (writable) channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/pro/linux-64 https://repo.anaconda.com/pkgs/pro/noarch https://conda.anaconda.org/obspy/linux-64 https://conda.anaconda.org/obspy/noarch package cache : /home/megies/anaconda/pkgs /home/megies/.conda/pkgs envs directories : /home/megies/anaconda/envs /home/megies/.conda/envs platform : linux-64 user-agent : conda/4.5.4 requests/2.13.0 CPython/2.7.11 Linux/4.9.0-6-amd64 debian/9 glibc/2.24 UID:GID : 1043:100 netrc file : /home/megies/.netrc offline mode : False ```

minrk commented 5 years ago

I believe this is fixed in more recent versions of the zeromq package (build 5), but we can add similar pin_run_as_build here, since we do link it directly.

megies commented 5 years ago

Ah, I see.. indeed it looks like the recent versions of the zeromq package (build 5) have the problem fixed.

Now I wonder why conda did not resolve to the highest available zeromq build number above, when updating pyzmq?

Coming from this env:

$ conda list --export | egrep '(pyzmq)|(zeromq)|(libsodium)'
libsodium=1.0.10=0
pyzmq=17.0.0=py27_3
zeromq=4.2.3=2

..if I request a newer pyzmq, I am offered zeromq build 4 which does not appropriately update libsodium:

$ conda install 'pyzmq>17.0'
Solving environment: done

## Package Plan ##

  environment location: /home/megies/anaconda/envs/default

  added / updated specs: 
    - pyzmq[version='>17.0']

The following packages will be UPDATED:

    pyzmq:  17.0.0-py27_3 conda-forge --> 17.1.2-py27hae99301_0 conda-forge
    zeromq: 4.2.3-2       conda-forge --> 4.2.5-hfc679d8_4      conda-forge

Proceed ([y]/n)? n

N.b. just updated to most recent conda just to make sure it's not an already fixed conda issue (conda info in details):

``` $ conda info active environment : default active env location : /home/megies/anaconda/envs/default shell level : 2 user config file : /home/megies/.condarc populated config files : /home/megies/.condarc conda version : 4.5.11 conda-build version : 3.7.1 python version : 2.7.11.final.0 base environment : /home/megies/anaconda (writable) channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/pro/linux-64 https://repo.anaconda.com/pkgs/pro/noarch https://conda.anaconda.org/obspy/linux-64 https://conda.anaconda.org/obspy/noarch package cache : /home/megies/anaconda/pkgs /home/megies/.conda/pkgs envs directories : /home/megies/anaconda/envs /home/megies/.conda/envs platform : linux-64 user-agent : conda/4.5.11 requests/2.13.0 CPython/2.7.11 Linux/4.9.0-6-amd64 debian/9 glibc/2.24 UID:GID : 1043:100 netrc file : /home/megies/.netrc offline mode : False ```

conda-forge / pyzmq-feedstock