Open richardartoul opened 7 years ago
meejah
commented
7 years ago Some of those packages are slightly out of date. I don't know how Python gets root certificates on Mac, but perhaps "pip install certifi" helps? If it does, then Python probably isn't finding the system root certificates properly.
richardartoul
commented
7 years ago Unfortunately the same problem persists after pip install certifi
meejah
commented
7 years ago There's a 17.2.0 for pyOpenSSL; does "pip install --upgrade autobahn" help? Does curl or a "normal" browser verify that site's certificate correctly? (FWIW, my browser on Linux does)
richardartoul
commented
7 years ago This problem doesn't occur with regular HTTPS requests (I.E this works fine):
import requests
requests.get('https://hyperdash.io/api/v1/sdk/wamp').contentLet me try upgrade
richardartoul
commented
7 years ago pip seems to think I'm up to date.
richie$ pip install --upgrade autobahn --no-cache-dir
Requirement already up-to-date: autobahn in ./env/lib/python2.7/site-packages
Requirement already up-to-date: six>=1.10.0 in ./env/lib/python2.7/site-packages (from autobahn)
Requirement already up-to-date: txaio>=2.7.0 in ./env/lib/python2.7/site-packages (from autobahn)richie$ pip install --upgrade pyopssl --no-cache-dir
Requirement already up-to-date: pyopenssl in ./env/lib/python2.7/site-packages
Requirement already up-to-date: six>=1.5.2 in ./env/lib/python2.7/site-packages (from pyopenssl)
Requirement already up-to-date: cryptography>=1.9 in ./env/lib/python2.7/site-packages (from pyopenssl)
Requirement already up-to-date: ipaddress in ./env/lib/python2.7/site-packages (from cryptography>=1.9->pyopenssl)
Requirement already up-to-date: asn1crypto>=0.21.0 in ./env/lib/python2.7/site-packages (from cryptography>=1.9->pyopenssl)
Requirement already up-to-date: enum34 in ./env/lib/python2.7/site-packages (from cryptography>=1.9->pyopenssl)
Requirement already up-to-date: idna>=2.1 in ./env/lib/python2.7/site-packages (from cryptography>=1.9->pyopenssl)
Requirement already up-to-date: cffi>=1.7 in ./env/lib/python2.7/site-packages (from cryptography>=1.9->pyopenssl)
Requirement already up-to-date: pycparser in ./env/lib/python2.7/site-packages (from cffi>=1.7->cryptography>=1.9->pyopenssl)
(env) richie$ python repo.py
connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)@meejah A little more digging seems to narrow this problem down to Twisted. I'm able to reproduce the issues as follows:
from __future__ import print_function
from twisted.internet import reactor
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
agent = Agent(reactor)
d = agent.request(
'GET',
'https://google.com',)
def cbResponse(ignored):
print('Response received')
d.addCallback(cbResponse)
def cbShutdown(ignored):
reactor.stop()
d.addBoth(cbShutdown)
reactor.run()In that snippet, "Response received" is never printed and the program just terminates. Wireshark logs show similar behavior with an Unknown CA error being thrown at the TLS level.
Replace the URL above with 'http://google.com' works fine though...
meejah
commented
7 years ago Hmm, okay. Thanks for the digging! That's ... weird.
The above snippet works for me on Debian with Twisted==17.5.0, pyOpenSSL==17.1.0, certifi==2017.4.17 so I'm going to guess this is Mac-only.
richardartoul
commented
7 years ago @meejah
What do you think of this? https://github.com/twisted/treq/issues/94#issuecomment-116226820
I'm not home to verify the suggested fix, but seems like this could be the issue.
meejah
commented
7 years ago What OS are you on?
richardartoul
commented
7 years ago OSX Sierra
meejah
commented
7 years ago Did you install Python with homebrew, or something else? (It seems like your Python or openSSL can't find any certificate roots at all). See the "note" about OS X here: https://twisted.readthedocs.io/en/twisted-17.5.0/core/howto/ssl.html
richardartoul
commented
7 years ago @meejah I figured out a solution, but its gross:
SSL_CERT_FILE="$(python -m certifi)" python test_script.pySolves the problem
but since I need a programatic solution...
import os
from certifi import where
os.environ['SSL_CERT_FILE'] = where()Also fixes it. At this point I'm probably just gonna ship it with this hack unless you have a better idea.
Also I'm happy to discuss how we can make this a better experience for other developers. I know its an issue with Twisted so its somewhat out of your hands, but maybe we can make Autobahn throw a more helpful error message or just do the environment hack I placed above?
meejah
commented
7 years ago See also: https://twistedmatrix.com/trac/ticket/9209
Looks like it might help to try "pip install certitude" instead, and/or install at least OpenSSL via homebrew. Did you install Python itself via homebrew or some other method? (In my past Mac experience, I found it much better to use the homebrew version of Python).
meejah
commented
7 years ago I'm leaving this open so that hopefully we can provide a better error (and/or some hints) for users.
@richardartoul can you confirm how you installed both Python and OpenSSL? (Basically: via homebrew or some other way?)
meejah
commented
7 years ago Also, it would be awesome if you can test an asyncio variant to see if you experience the same problem!
richardartoul
commented
7 years ago @meejah I'm actually not sure how I setup Python/OpenSSL on this machine. I've been writing software on it for 2 years so its possible I've reinstalled them multiple times and messed up the configuration. Anecdotally, two of my friends also have macbooks and it worked on one of their machines and not on the other.
If there are some commands you want me to run I'm happy to do it, but I probably won't dig into that too much further because I'm writing a library that needs to run on all machines and I won't be able to tell people to reinstall python or OpenSSL so I'll have to live with this certifi hack for now.
meejah
commented
7 years ago I'm just trying to learn what's special about how a "not working" machine is set up so hopefully we can steer new-installers in the right direction.
richardartoul
commented
7 years ago Sure, is there an easy way for me to check how I setup Python/OpenSSL?
meejah
commented
7 years ago Does which python or locate libssl turn up any hints?
oberstet
commented
7 years ago @richardartoul you could also try our Docker images for Autobahn. That way, you should be shielded from issues like this ..
meejah
commented
7 years ago For the record (as per the Twisted bug 9209 linked above), our install instructions for non-Docker (etc) users should tell people to use "homebrew" to install Python on Mac OS if they want TLS root certificates to "probably work".
richardartoul
commented
7 years ago Inside of virtual environment
(env)Richards-MacBook-Pro:test richie$ which python
/Users/richie/Documents/projects/test/env/bin/python
(env)Richards-MacBook-Pro:test richie$ which openssl
/Users/richie/anaconda3/bin/opensslOutside of virtual environment
Richards-MacBook-Pro:test richie$ which python
/Users/richie/anaconda3/bin/python
Richards-MacBook-Pro:test richie$ which openssl
/Users/richie/anaconda3/bin/opensslSo looks like it doesn't work with anaconda, and also doesn't seem to work with the python installation in virtual environment either. I actually don't know how virtualenv works under the hood, is that symlink to system python or its own installation?
richardartoul
commented
7 years ago It doesn't seem like just telling people to use Docker is really a good enough solution though, in our case for example, we're developing a library for other people to run on their own machines.
Could we catch this error with a errback call where the Autobahn code calls into twisted?
richardartoul
commented
7 years ago @meejah I got this error when I ran locate libssl
WARNING: The locate database (/var/db/locate.database) does not exist.
To create the database, run the following command:
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.locate.plist
Please be aware that the database can take some time to generate; once
the database has been created, this message will no longer appear.Running the suggest command didn't fix it
oberstet
commented
7 years ago we're developing a library for other people to run on their own machines
What kind of machines (developer boxes or embedded devices)?
Curious: what library?
Could we catch this error with a errback call where the Autobahn code calls into twisted?
Yep, that would be definitely good. I changed the issue title to reflect that.
richardartoul
commented
7 years ago This is it: https://github.com/hyperdashio/hyperdash-sdk-py
The website's not up yet, but the idea is that it automatically monitors your machine learning jobs.
We originally stared with websockets and then realized that we had no way to signal back to the SDK that we failed to process a message so we're trying WAMP now for the RPC features
EDIT: Developer boxes
oberstet
commented
7 years ago Interesting! Thanks for sharing. Yeah, the combination of RPC+PubSub should bring you more flexibility at higher level of abstraction compared to raw WebSocket.
Also, should you use Crossbar.io as a WAMP router, you get more goodies for free, like the HTTP/REST bridge. I once needed to access data exposed in a registered WAMP procedure from Jupyter. Easy using the HTTP/REST bridge: I could call and fetch the data directly from a code snipped in my Jupyter workbook. Publishing is easy too. Registering/subscribing isn't obvious though.
Developer boxes: actually, personally I am pretty bullish on using Docker (also) for development. Eg I don't have a Java SDK on my notebook, all the toolchain is in a Docker toolchain container. Only thing I do on the host (notebook) is edit source code. So I can work on AutobahnJava without loosing more hair. It effectively shields me from all the details getting a complete environment and toolchain set up correctly. And I am doing this despite being on Ubuntu, which I guess has less issues (for a developer) compared to OSX/Windows. Like this TLS cert stuff. On the other hand, Docker experience on non-Linux is still lagging behind too. This toolchain Docker image approach saves me countless hours ..
richardartoul
commented
7 years ago Yeah I'll definetly look into the crossbar.io WAMP router at some point. Currently we're just using the Turnpike Golang library, but I'd like to dig into the official router some more and see what we could use it for.
Agreed I think Docker is great and we use it for development/production of our server, however, our target demographic for the library is data scientists who are already happy with their setup, but just want to be able to remotely monitor their jobs. Very smart people, but we're trying to give them a python package to import and forget about, not ask them to install docker.
I've also never heard of anyone running a deep learning job in a docker container, but I'm not an expert haha
oberstet
commented
7 years ago Ok, if your audience is more data scientists, they probably expect something easy working on Anaconda.
FWIW, even Anaconda seems to be pushing Docker: https://www.continuum.io/blog/developer-blog/anaconda-and-docker-better-together-reproducible-data-science
Eg this gets me started in like 1 minute
docker run -i -t -p 8888:8888 continuumio/anaconda3 \
/bin/bash -c "/opt/conda/bin/conda install jupyter -y \
--quiet && mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook \
--notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser --allow-root"I had to add
--allow-rootcompared to above blog post.
meejah
commented
7 years ago Packaging Python applications for "end users' systems" still doesn't have as good a story as I would like :/
Docker is pretty good. "snapcraft"/snaps from Canonical and "flatpak" both sound very promising, though. There's also a tool called AppImage but it doesn't provide sandboxing like the first two. I haven't spent a lot of time looking into either of these very extensively, but they do at least aim to provide easy installation of "apps" on Linux systems.
oberstet
commented
7 years ago I think snaps are really promising. Also because it takes OS updates into account (an Ubuntu Core system completely consists of snap .. including kernel and OS userland).
snaps also run on most Linux distros these days
https://snapcraft.io/docs/core/install
However, the isolation only works with Ubuntu AppArmor enabled kernels currently (as far as I understand, not all Ubuntu AppArmor kernel mods are upstreamed yet).
Crossbar.io is currently available as a snap for x86-64 and armhf - aarch64 doesn't work yet in snapcraft.
meejah
commented
7 years ago Yeah, the sandboxing of flatpak uses SElinux (via 'bubblewrap') and works on more linux distros than just Ubuntu. However, I don't have a good way to compare them beyond that ;)
probonopd
commented
7 years ago Providing an AppImage would have, among others, these advantages:
appimagedHere is an overview of projects that are already distributing upstream-provided, official AppImages.
richardartoul
commented
7 years ago I understand the benefits of alternative packaging methods, but unfortunately I'm writing a pip installable library, not a stand-alone executable
Hey ya'll,
This code works fine when I connect to localhost with ws://, but it fails when I try to use it with wss:// and a production server with a valid certificate. Specifically, my TLS certificate seems to get rejected (CA unknown) despite being valid. I've posted wireshark logs and a minimum reproduction below.
Here is the minimum reproduction:
This generates logs like this:
Executing pip freeze shows these dependencies:
In a python REPL:
I thought maybe it was related to an old version of openssl, but when I run the code using Conda which has a newer version:
conda REPL:
I still get the same error.
Any thoughts? The wireshark logs seem to indicate that thats a TLS issue (specifically, the CA unknown error), however, I'm pretty sure our certificate is fine:
autobahn_wamp_packets.pcapng.zip