pyzmq should use Anaconda's zeromq on Windows

[chrisjbillington]

It appears that on Windows Anaconda pyzmq is using the libzmq bundled with it rather than the one installed by Anaconda:

conda update conda
conda create -n testing python=2.7
activate testing
conda install zeromq
conda install pyzmq
conda list

(Possibly unrelated, but the above commands have the effect of updating Python itself from 2.7 to 3.6, I'm not sure why, I'm pretty sure I was able to produce a similar situation whilst remaining on Python 2.7, but the above is to reproduce for the sake of this issue)

# packages in environment at C:\Anaconda2\envs\testing:
#
certifi                   2018.1.18                py36_0
libsodium                 1.0.15               h3c5a7f5_0
pip                       9.0.1                    py36_5
python                    3.6.4                h0c2934d_3
pyzmq                     17.0.0           py36hfa6e2cd_0
setuptools                38.5.1                   py36_0
vc                        14                   h0510ff6_3
vs2008_runtime            9.00.30729.1         hfaea7d5_1
vs2015_runtime            14.0.25123                    3
wheel                     0.30.0           py36h1201a8d_1
wincertstore              0.2              py36hcdd9a18_0
zeromq                    4.2.3                hd6b2f15_3

Python 3.6.4 |Anaconda, Inc.| (default, Mar 12 2018, 20:25:16) [MSC v.1900 32 bi
t (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq
>>> zmq.__version__
'17.0.0'
>>> zmq.zmq_version()
'4.1.6'
>>>

Using Anaconda's libzmq, which is built with libsodium, is preferable because it allows faster cryptography compared to tweetnacl (the crypto library bundled with the version of libzmq bundled with pyzmq).

[mingwandroid]

R and all of its packages need to be built with GCC on Windows instead of msvc. For this reason we use MSYS2 packages.

[mingwandroid]

R and all of its packages need to be built with GCC on Windows instead of msvc. For this reason we use MSYS2 packages.

[chrisjbillington]

In that case, since the pyzmq package comes with its own libzmq that it was built against, could it come with libsodium too? There needs to be two copies of zeromq - one for R, one for python (the latter contained within the pyzmq install dir), so why not two copies of libsodium?

Zeromq devs strongly recommend libsodium for encryption rather than tweetnacl, regardless of the speed thing which is the only reason I noticed they were different.

[mingwandroid]

Sorry I totally misread this issue! You are only talking about Python here.

OK, in my experience, zeromq on Windows has been in a pretty buggy state for a good while now, but I've not been able to make a good reproduction case to report upstream. It seems to be a race condition managing some handles.

For this reason we've had to play it very safe with pyzmq on Windows. Reopening as it's definitely something we should aim to fix.

[sigiesec]

@mingwandroid Unfortunately, there were some bugs in both the select-based and poll-based poller implementations on Windows, which were only fixed after the latest release (4.2.3), i.e. the fixes are only in 4.2.4-unreleased for now:

• https://github.com/zeromq/libzmq/pull/2939
• https://github.com/zeromq/libzmq/pull/2943

Maybe these are related to the issues you experienced.

[mingwandroid]

@sigiesec thank you very much for this information. When is the next release planned?

[sigiesec]

4.2.4 should be out in the next few days.

[sigiesec]

libzmq 4.2.4 has been released: https://github.com/zeromq/libzmq/releases/tag/v4.2.4

[mingwandroid]

@sigiesec thank you, I am working on updating to this version now. Initial tests look good. It's great to have a reliable zeromq on Windows again.

[makiolo]

I have this solved in Python 2.7:

get tag v19.0.2 from pyzmq

git clone https://github.com/zeromq/pyzmq.git
cd pyzmq
git checkout v19.0.2

Install zeromq==4.3.2 and cython (branch 0.20.x)

conda install conda-forge::zeromq=4.3.2
pip install cython==0.29.37

In pyzmq source code create a setup.cfg with this code. In ths example, I have installed Anaconda in C:\Miniconda2

[global]
zmq_prefix = C:/Miniconda2/Library

[build_ext]
library_dirs = C:/Miniconda2/Library/lib
include_dirs = C:/Miniconda2/Library/include

And now type (You need Visual C++ for Python 2.7):

SET DISTUTILS_USE_SDK=1
SET MSSdk=1
call "C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\9.0\vcvarsall.bat" amd64
pip install --ignore-installed --install-option="--zmq=C:\Miniconda2\Library" .

Now you have installed pyzmq using zeromq from Anaconda.