inform about ALPN/NPN support on local/remote

[mylh]

Hello,

I encountered the following issue when connect to Apple push notification service with hyper. Current APNS endpoint uses HTTP/2. Connects from python 2.7.10 works good, but when I try to connect from python 3.4 I just get ConnectionReset error every time. After spending some hours for investigation I ended up with the following. APNS does not support NPN (which is considered obsolete as I understand) and only support ALPN. And ALPN support in python ssl module is only started from python 3.5. In hyper.tls we have the following code

    # ALPN is newer, so we prefer it over NPN. The odds of us getting different
    # answers is pretty low, but let's be sure.
    with ignore_missing():
        proto = ssl_sock.selected_alpn_protocol()

    with ignore_missing():
        if proto is None:
            proto = ssl_sock.selected_npn_protocol()

What we see here is that absense of ALPN support is silently ignored. And since NPN is not supported at endpoint hyper follows http1.1 connection scheme though current endpoint is HTTP2.

I think that it is better to allow developers using hyper to specify wether they want to ignore_missing or not and raise exceptions when this happens. At least they will not spent a day figuring out why it works in one python version and not in other. And most of the times you know when HTTP/2 is supported on your endpoint but even if you use HTTP20Connection you still get assertion error because NPN is not supported at remote end and ALPN at your end and this goes silently.

What do you think?

(looks like for my case I will need to use local copy of the hyper with hardcoded 'h2' proto by default, and I'm not happy about it)

[Lukasa]

@mylh Yeah, this is admittedly a concern. The debuggability of it is a real issue, but it's mildly problematic to resolve in a way that doesn't have some really unfortunate usability problems:

• Either we add a configuration flag for this, which is a slightly tricky thing to do in a way that is clear.
• We can assert in situations where neither NPN or ALPN are available (not really ideal, and wouldn't have fixed this problem as NPN is available)

I'm aware this is an issue though, and have been considering for a little while whether we should extend ignore_missing to log at warning level in situations when one of the two protocols is not present.

As to what to do in your specific case: where are you getting your Python from? It would obviously be ideal if you could use either Python 2.7.9+ or 3.5+, either of which does have support for ALPN in the standard library. Unfortunately, if you're on Python 3.4 you're in a tricky middle stage where currently hyper uses the standard library because NPN is available, rather than falling back and requiring PyOpenSSL.

We also have a patch that is currently languishing (#197) that basically just lets you override what hyper thinks is going on. That patch is probably suitable for your use case, so you may want to run a local copy of hyper with that patch applied until such time as I'm able to ship an update to hyper that includes it.

How does that seem to you?

[mylh]

Well I should search for APNS in the repo :) I think I would go with the patch. The problem arises with Ubuntu 14.04 where python is in 2.7.6 and 3.3. and this is the default Ubuntu LTS release now :( I will think about a way to implement this in hyper considering your input. If I have ideas will definitely share

[Lukasa]

Arg, yeah, the Ubuntu LTS releases are a problem. You'd find that even with newer Pythons your OpenSSL is too old, and doesn't support ALPN anyway (ALPN was added in OpenSSL 1.0.2). So you really have no option but to just circumvent it.

[yuvadm]

I was getting the exact same problem on Python 3.4 on Heroku. Luckily I can simply bump up the runtime. So now when testing on 3.5.1 I'm still getting:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/app/.heroku/python/lib/python3.5/site-packages/requests/sessions.py", line 511, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/app/.heroku/python/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/app/.heroku/python/lib/python3.5/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/app/.heroku/python/lib/python3.5/site-packages/hyper/contrib.py", line 68, in send
    resp = conn.get_response()
  File "/app/.heroku/python/lib/python3.5/site-packages/hyper/common/connection.py", line 124, in get_response
    return self._conn.get_response(*args, **kwargs)
  File "/app/.heroku/python/lib/python3.5/site-packages/hyper/http11/connection.py", line 198, in get_response
    self._sock.fill()
  File "/app/.heroku/python/lib/python3.5/site-packages/hyper/common/bufsocket.py", line 172, in fill
    raise ConnectionResetError()
ConnectionResetError

Any configuration I need to change?

[Lukasa]

In this situation the connection is getting forcibly closed. The problem here seems to be that hyper is sending a HTTP/1.1 request, presumably because either you don't have ALPN available (and so no protocol was negotiated) or because you used a non-https URL.

The version of hyper in the master branch allows you to tell hyper to ignore the ALPN negotiation and to assume a given result, but by default you won't be able to use that with the requests adapter provided in contrib. I'd welcome a patch to fix that, at which point you should be able to use the master branch until such time as I draft a release.

[yuvadm]

@Lukasa thanks for the quick response! Any idea why ALPN isn't available here despite using python 3.5.1?

Indeed I'm getting this locally

>>> import ssl
>>> ssl.HAS_ALPN
True

And this on heroku

>>> import ssl
>>> ssl.HAS_ALPN
False

Could this be Heroku using OpenSSL 1.0.2? I'd rather use a buildpack that just uses ALPN rather than go through the patching process.

[Lukasa]

So ALPN is only available on newer OpenSSLs, as you've discovered. My bet is that Heroku by default uses Ubuntu 14.04 for its base OS, which contains OpenSSL < 1.0.2: sadly, ALPN was introduced in OpenSSL 1.0.2.

I'm not sure how troublesome it will be to provide yourself with a new OpenSSL on Heroku, but the answer is probably "somewhat troublesome".

[yuvadm]

@Lukasa that's what i'm trying to figure our right now. I've built custom buildpacks before, but need to refresh my memory. I'll update here for posterity once I figure it out :) thanks!

[matangover]

For anyone else coming here looking how to run Python with ALPN support (OpenSSL 1.0.2) on Heroku, see instructions here: Compiling Python with custom OpenSSL on Heroku. Hope this helps :)

[Lukasa]

Thanks for that note @matangover! :sparkles:

[Lec2cn]

@Lukasa Hey bro. I am using 0.7 hyper for APNS services.How can i ignore the ConnectionResetError error to continue using hyper? Thanks a lot. -- Why i am doing this , it's because I got a ValueError: Could not unserialize key data. .I've been searching for the entire Internet that how can i update the openssl version in the Ubuntu.When i done that , i can't let python use the new version openssl (Although i change the setup.dist in the python building directory , don't know which part did wrong). So i didn't use the latest version of the openssl.And come up with this problem.

Thanks for listening my issue here..

[Lukasa]

You can't ignore a ConnectionResetError: that occurs because the connection was forcefully closed by the remote peer. How are you hitting the problem?

[Lec2cn]

@Lukasa I am using a Python2.7 which output ssl version OpenSSL 1.0.1f 6 Jan 2014 When i using the .p8 file provided by Apple , send the request through hyper , the cryptography error raise up. (https://github.com/pyca/cryptography/issues/3232). So i break the .p8 lines then it works. but now i am facing the problem i describe.

[Lukasa]

No, the problem is yours: OpenSSL 1.0.1f does not support ALPN, so the negotiation fails. You'll need to install OpenSSL 1.0.2 or later.

[Lec2cn]

@Lukasa Solving the problem by upgrade ubuntu to the new version and the openssl automatically update to 1.0.2. Thanks for helping!