Regression: Version 16.1 doesn't work with Kodi

[candrews]

I'm a user of Kodi, not a developer, so I apologize ahead of time for the lack of detail in this bug report.

Versions 15.1 and 16.0 work fine. Version 0.16.1 does not work with Kodi. I've reported the issue to Kodi at http://trac.kodi.tv/ticket/16914

In short, when you try to perform an operation in Kodi that's implemented using urllib2, such as playing a Youtube video, playback fails, this error is sent to stdout:

extern "Python": function Cryptography_rand_bytes() called, but @ffi.def_extern() was not called in the current subinterpreter.  Returning 0.

and this stack trace is logged:

23:06:45 T:139931166926592   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <class 'urllib2.URLError'>
                                            Error Contents: <urlopen error [Errno 0] Error>
                                            Traceback (most recent call last):
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/default.py", line 7, in <module>
                                                runner.run(__provider__)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/kodion/runner.py", line 32, in run
                                                __RUNNER__.run(provider, context)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/kodion/impl/xbmc/xbmc_runner.py", line 23, in run
                                                results = provider.navigate(context)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/kodion/abstract_provider.py", line 123, in navigate
                                                result = method(context, re_match)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/kodion/register_provider_path.py", line 12, in wrapper
                                                return func(*args, **kwargs)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/youtube/provider.py", line 367, in on_play
                                                return yt_play.play_video(self, context, re_match)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/youtube/helper/yt_play.py", line 17, in play_video
                                                video_streams = client.get_video_streams(context, video_id)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/youtube/client/youtube.py", line 80, in get_video_streams
                                                video_streams = video_info.load_stream_infos(video_id)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/youtube/helper/video_info.py", line 330, in load_stream_infos
                                                return self._method_get_video_info(video_id)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/youtube/helper/video_info.py", line 542, in _method_get_video_info
                                                result = requests.get(url, params=params, headers=headers, verify=False, allow_redirects=True)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/kodion/simple_requests/api.py", line 179, in get
                                                return _request('GET', url, **kwargs)
                                              File "/home/mythtv/.kodi/addons/plugin.video.youtube/resources/lib/kodion/simple_requests/api.py", line 155, in _request
                                                response = opener.open(request)
                                              File "/usr/lib64/python2.7/urllib2.py", line 429, in open
                                                response = self._open(req, data)
                                              File "/usr/lib64/python2.7/urllib2.py", line 447, in _open
                                                '_open', req)
                                              File "/usr/lib64/python2.7/urllib2.py", line 407, in _call_chain
                                                result = func(*args)
                                              File "/usr/lib64/python2.7/urllib2.py", line 1241, in https_open
                                                context=self._context)
                                              File "/usr/lib64/python2.7/urllib2.py", line 1198, in do_open
                                                raise URLError(err)
                                            URLError: <urlopen error [Errno 0] Error>
                                            -->End of Python script error report<--

[hynek]

I see no pyOpenSSL in your traceback; urllib2 uses the standard library's ssl module. But there seems to be a call to an uninitialized cryptography? Sadly there’s no details why or by whom it is called? Maybe @reaperhulk can make sense of this but from my vantage point pyOpenSSL isn’t really involved. Is it used by kodi at all?

[tiran]

It smells like a problem with mod_wsgi subinterpreters and cryptography's random engine hook.

[candrews]

Kodi isn't a web application and doesn't use mod_wsgi.

As for how Kodi uses pyopenssl, I don't know - but, I do know that if one uses a version of pyopenssl before 16.1 (I tested with 15.1 and 16.0) Kodi works fine. So it has to be a regression in pyopenssl 16.1 because that is the only variable.

I can guess that perhaps the problem is that pyopenssl configures openssl different in 16.1 than in previous versions and there's a threading issue causing that configuration to be applied to urllib2 (which uses openssl but not via pyopenssl).

[reaperhulk]

I've looked at Kodi's source before and it is a Python subinterpreter problem. It's possible we stopped disabling cryptography's osrandom engine in pyopenssl 16.1? @candrews inside the youtube video plugin if you can find where it imports from OpenSSL (probably from OpenSSL import SSL) add these two lines after that import:

from cryptography.hazmat.backends.openssl.backend import backend
backend.activate_builtin_random()

[alex]

I don't recall pyOpenSSL disabling the osrandom engine.

On Sun, Sep 18, 2016 at 12:57 PM, Paul Kehrer notifications@github.com wrote:

I've looked at Kodi's source before and it is a Python subinterpreter problem. It's possible we stopped disabling cryptography's osrandom engine in pyopenssl 16.1? @candrews https://github.com/candrews inside the youtube video plugin if you can find where it imports from OpenSSL (probably from OpenSSL import SSL) add these two lines after that import:

from cryptography.hazmat.backends.openssl.backend import backend backend.activate_builtin_random()

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pyca/pyopenssl/issues/542#issuecomment-247859346, or mute the thread https://github.com/notifications/unsubscribe-auth/AAADBCCAUn-jyTF-tJMvY3aAmVdiNoAHks5qrW1tgaJpZM4J_yZu .

"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[reaperhulk]

@alex originally we never imported the backend at all, just the binding. So we never enabled the engine.

[alex]

Ah, so now that we import the backend from to_cryptography/from_cryptography it's triggered.

On Sun, Sep 18, 2016 at 12:58 PM, Paul Kehrer notifications@github.com wrote:

@alex https://github.com/alex originally we never imported the backend at all, just the binding. So we never enabled the engine.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pyca/pyopenssl/issues/542#issuecomment-247859413, or mute the thread https://github.com/notifications/unsubscribe-auth/AAADBMWXEkGI0qNZKJN9RyHQ2B8thnUXks5qrW2sgaJpZM4J_yZu .

"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[reaperhulk]

Yep. Damn global side effects.

[hynek]

Any ideas how to proceed? Simplest fix would be to move the imports into those functions and release a 16.1.1 I guess?

[reaperhulk]

We just need to call backend.activate_builtin_random() after the import in crypto.py.

[alex]

Uhhh, sounds bad, I would definitely be opposed to a patch that did that.

[reaperhulk]

We intentionally didn't want to have pyopenssl use the osrandom engine in the past. This is why the binding doesn't activate it, only the backend. Now things are trickier since we're using higher levels of cryptography's APIs inside pyopenssl. Disabling osrandom is unfortunate, but we've repeatedly run into concrete breakage due to use of subinterpeters. It's not hard to find persistent suggestions in various forums that people downgrade their cryptography versions to avoid these bugs, which is pretty terrible. With this change we've created the same problem in pyopenssl and given users of tools like kodi a strong incentive to pin at 16.0 forever.

One solution would be to go back to a C based osrandom, but you've strongly resisted that option in the past due to the increase in complexity it requires. I'd suggest that we can't have our cake and eat it too any more. We need to do something distasteful -- either we hoist osrandom back into C or we disable osrandom engine (first in pyopenssl but also ultimately in cryptography). Given that the latter poses major risk to any user forking processes it seems like there's really only one way forward.

On Sep 20, 2016, at 8:59 AM, Alex Gaynor notifications@github.com wrote:

Uhhh, sounds bad, I would definitely be opposed to a patch that did that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[alex]

I'd prefer moving the osrandom engine back to C code.

If we put the switch_back_to_builtin code in pyOpenSSL, it means if you use both pyOpenSSL and cryptography you get the opposite behavior you do today.

On Tue, Sep 20, 2016 at 10:13 AM, Paul Kehrer notifications@github.com wrote:

We intentionally didn't want to have pyopenssl use the osrandom engine in the past. This is why the binding doesn't activate it, only the backend. Now things are trickier since we're using higher levels of cryptography's APIs inside pyopenssl. Disabling osrandom is unfortunate, but we've repeatedly run into concrete breakage due to use of subinterpeters. It's not hard to find persistent suggestions in various forums that people downgrade their cryptography versions to avoid these bugs, which is pretty terrible. With this change we've created the same problem in pyopenssl and given users of tools like kodi a strong incentive to pin at 16.0 forever.

One solution would be to go back to a C based osrandom, but you've strongly resisted that option in the past due to the increase in complexity it requires. I'd suggest that we can't have our cake and eat it too any more. We need to do something distasteful -- either we hoist osrandom back into C or we disable osrandom engine (first in pyopenssl but also ultimately in cryptography). Given that the latter poses major risk to any user forking processes it seems like there's really only one way forward.

On Sep 20, 2016, at 8:59 AM, Alex Gaynor notifications@github.com wrote:

Uhhh, sounds bad, I would definitely be opposed to a patch that did that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pyca/pyopenssl/issues/542#issuecomment-248313480, or mute the thread https://github.com/notifications/unsubscribe-auth/AAADBGJrPyi9HlICjSD_Lx0z6BDFntMdks5qr-oWgaJpZM4J_yZu .

"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[reaperhulk]

This should be resolved in 16.2.0 (a workaround for now, osrandom improvements in the future)

[effemmeffe]

As a kodi user I can confirm that the problem is still there. My kodi runs on ubuntu 16.04, following instruction here: https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/1636573 I upgraded to packages python-cryptography_1.7.1-2_amd64.deb and python-openssl_16.2.0-1_all.deb and did sudo easy_install --upgrade PyOpenSSL but the file .xsession-errors is still full of messages like this. I tried to truncate the file to 0 as a workaround, but it seems it doesn't work as I was expecting, so I need to rebbot the machine now and then to have the file clear.

[hynek]

Is there a way for your to verify what versions Kodi is actually using? You seem to be mixing packages from multiples sources and it’s hard to guess, which are actually used.

[effemmeffe]

I have a plain 16.04 version of ubuntu, afaik the kodi that comes with it is this: https://launchpad.net/~team-xbmc/+archive/ubuntu/ppa I don't know if there is a way to know the kodi version via cli, I can't check on my TV right now. The only thing I mixed is those two packages taken from the next version of ubuntu.

[Rudd-O]

The problem continues. I tried pyOpenSSL 16 (stock in Fedora) and pyOpenSSL 16.2 (which I compiled myself). No difference. Errno 0. That's absurd, how can an error have errno zero?

[Rudd-O]

For the record, this is the version of cryptography I am using (derived directly from the file that Kodi has open in question):

python2-cryptography-1.5.3-3.fc25.x86_64

You can find this package in the Fedora Koji system if you need to inspect its contents.

[Rudd-O]

Actually, incorrect. Downgrading doesn't fix it.

[hynek]

That indicates strongly that Kodi is not picking up your pyOpenSSL installations.

[Rudd-O]

It is. I confirmed by hacking on the source of the Kodi plugin that uses urllib2, and it is picking the distribution's pyOpenSSL (16 or 16.2, either of them I tried). I looked at the __file__ attribute of the module. It's picking up the expected module. No difference.

Here's what makes me think it's a problem with either urllib2 or pyOpenSSL:

If I import the OpenSSL module at the top of the source file of the Kodi plugin, then urllib2 stops spewing the Errno 0 errors.

So it's a side effect somewhere, related to loading the pyOpenSSL module later than at top-of-module-source. When it loads early (before urllib2 makes its first request) everything works fine.

I've confirmed this is the case. Check the referred bug one comment above yours for a patch that "fixes" the problem with the plugin by loading pyOpenSSL before urllib2.

[MarkCallow]

Is this fixed in Ubuntu 17.04?

I applied one of the workarounds I found on the web to 16.04 and got things working but a subsequent regular software update appears to have reverted the fix because the problem is happening again. I'm fed up so considering a distribution upgrade.

[reaperhulk]

I don't know what versions are packaged with Ubuntu but this issue is fully resolved by upgrading cryptography to 1.7 or better and is also likely to be resolved by upgrading to pyOpenSSL 16.2 or better. The latter is just a workaround, the cryptography upgrade fixes the underlying issue.

[MarkCallow]

Good news! Ubuntu 17.04 has python-cryptography 1.7.1-2 and all Kodi add-ons are updating correctly.

[Moini]

Is it possible to fix this for Ubuntu 16.04, too?

[reaperhulk]

You'll have to update cryptography yourself. Ubuntu won't backport the package update.

[Moini]

Ah, thank you for letting me know, @reaperhulk !

The other option then is to update my Linux Mint... probably in October. I can wait until then.