search

EFForg / https-everywhere

A browser extension that encrypts your communications with many websites that offer HTTPS but still allow unencrypted connections.
https://eff.org/https-everywhere
Other
3.36k stars 1.09k forks source link

Error running fetch-test.sh: "Unknown cipher in list" #5628

Closed jeremyn closed 8 years ago

jeremyn commented 8 years ago

I'm getting Fetch error messages such as you see below when I run fetch-test.sh against any ruleset I've tried so far. The example I've given below is run with commit 45083c6da8daeac07e30c154974566d770446b32. This is using Vagrant with a Windows 10 host/Fedora 24 Cloud guest (see https://github.com/EFForg/https-everywhere/pull/5584). bash test.sh seems to finish without error.

What's causing this error and how can I fix it? Thanks.

[vagrant@localhost vagrant]$ ./fetch-test.sh src/chrome/content/rules/Amnesty_USA.org.xml
INFO Finished comparing http://amnestyusa.org/ -> https://amnestyusa.org/. Rulefile: src/chrome/content/rules/Amnesty_USA.org.xml.
INFO Finished comparing http://www.amnestyusa.org/ -> https://www.amnestyusa.org/. Rulefile: src/chrome/content/rules/Amnesty_USA.org.xml.
ERROR src/chrome/content/rules/Amnesty_USA.org.xml: Fetch error: http://amnestyusa.org/ => https://amnestyusa.org/: (59, 'Unknown cipher in list: RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-S')
ERROR src/chrome/content/rules/Amnesty_USA.org.xml: Fetch error: http://www.amnestyusa.org/ => https://www.amnestyusa.org/: (59, 'Unknown cipher in list: RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-S')
INFO Finished in 0.29 seconds. Loaded rulesets: 1, URL pairs: 2.
[vagrant@localhost vagrant]$
fuglede commented 8 years ago

The tests use pycurl, so a first guess would be that pycurl or (more likely) one of its dependencies are outdated on your setup. Here's what I see on a system which normally runs the tests without problems:

In [1]: import pycurl

In [2]: pycurl.version
Out[2]: 'PycURL/7.19.5.3 libcurl/7.38.0 OpenSSL/1.0.1t zlib/1.2.8 libidn/1.29 libssh2/1.4.3 librtmp/2.3'

Can you check the corresponding versions on your system?

jeremyn commented 8 years ago

Sure. This is on Windows 10/Fedora 24 Cloud/Vagrant. I had to separately install pycurl because the default pycurl from pip did not work (see pull request #5584).

Output:

>>> import pycurl
>>> pycurl.version
'PycURL/7.43.0 libcurl/7.47.1 NSS/3.23 Basic ECC zlib/1.2.8 libidn/1.32 libpsl/0.13.0 (+libidn/1.32) libssh2/1.7.0 nghttp2/1.7.1'

The main difference between our configurations seems to be my NSS vs your OpenSSL, which makes sense given the problem I documented in the pull request.

jeremyn commented 8 years ago

Also, I'm able to use pycurl to fetch https://amnestyusa.org/ in the Python interpreter using the Python 2 directions at http://pycurl.io/docs/latest/quickstart.html .

fuglede commented 8 years ago

I see. We've had similar issues with gnutls; I can see how that's annoying but something you can always do would be to double-check the problematic hosts, and let Travis carry out its OpenSSL based tests afterwards.

I'm not sure we can do much else here. Perhaps have the script output a specific error in case pycurl uses libraries that have issues. @Hainish?

jeremyn commented 8 years ago

I dug into this some more. Also, I'm checking this with Fedora 24 Workstation using a GUI in VirtualBox, instead of using Vagrant, just to simplify things.

Hardcoded in test/rules/src/https_everywhere_checker/http_client.py is a list of default ciphers, currently:

RC4-MD5:RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256

These ciphers are ultimately passed to pycurl. The error Unknown cipher in list means that pycurl sees a cipher in this list that it doesn't recognize. The cipher list logged with the error (by this code) is a truncated version of the above list.

The problem is that NSS and OpenSSL use different, possibly incompatible names for any given cipher, assuming they both support that cipher. I'm guessing a little here, but as an example, consider what OpenSSL labels RC4-MD5. If you look at this list in the Mozilla Servo repository, it looks like RC4-MD5 matches TLS_RSA_WITH_RC4_128_MD5 among others. (I have no idea how reliable this Servo list is, by the way.) curl documentation for CURLOPT_SSL_CIPHER_LIST here leads us to mod_nss documentation here. (The content type for the mod_nss page is text/plain so you might need to save it to your desktop and then open the file with your browser to see rendered HTML.) In the mod_nss documentation we see that TLS_RSA_WITH_RC4_128_MD5 is labeled rsa_rc4_128_md5.

So if we're using NSS instead of OpenSSL, we need to tell pycurl to use rsa_rc4_128_md5 instead of RC4-MD5. Lucky for us, we can override the cipherList by adding the following line to the [http] section of test/rules/manual.checker.config:

cipherList = rsa_rc4_128_md5

The code that enables this override is here. With this option set, we no longer get Unknown cipher in list when running ./fetch-test.sh src/chrome/content/rules/Amnesty_USA.org.xml. Of course, I guess this only works because https://amnestyusa.org/ happens to support rsa_rc4_128_md5.

So a possible fix for this issue is to provide an alternate default cipherList for NSS, corresponding to the existing default list for OpenSSL, and to somehow switch to the new cipherList when using NSS instead of OpenSSL. If you agree this is an appropriate fix, please let me know and I can work on it.

fuglede commented 8 years ago

Thanks a lot for the digging! Ultimately what is appropriate and what is not is not for me to decide, but that certainly sounds like a reasonable approach to me.

Hainish commented 8 years ago

I'm inclined to just standardize on OpenSSL for testing. When we were testing in travis' build environment rather than a full travis container, we had to specify pycurl to build with OpenSSL support. It seems to me that this is the most appropriate solution, and the one that's likely to give us the most standardized results across all dev environments.

@jeremyn I suggest for development & testing to use docker and follow the standard steps for testing specified in .travis.yml. Barring that, you can build pycurl with OpenSSL in virtualenv using the steps here: http://pycurl.io/docs/latest/install.html

jeremyn commented 8 years ago

@Hainish pycurl is an interface to libcurl, and both need to use the same SSL backend. On Red Hat systems, libcurl uses the NSS SSL backend. libcurl is also a dependency of dnf and rpm, so it can't be removed or modified as far as I know. So requiring OpenSSL means that you don't support development on Red Hat systems. Is that right?

jeremyn commented 8 years ago

Also, for what it's worth, this seems to be a maximal cipherList based on the mod_nss documentation linked above, pulled from the table under NSSCipherSuite and removing some ciphers that were not recognized on Fedora 24:

cipherList = rsa_3des_sha,rsa_des_sha,rsa_null_md5,rsa_null_sha,rsa_rc2_40_md5,rsa_rc4_128_md5,rsa_rc4_128_sha,rsa_rc4_40_md5,fips_des_sha,fips_3des_sha,rsa_des_56_sha,rsa_rc4_56_sha,rsa_aes_128_sha,rsa_aes_256_sha,rsa_aes_128_gcm_sha_256,dhe_rsa_aes_128_gcm_sha_256,ecdh_ecdsa_null_sha,ecdh_ecdsa_rc4_128_sha,ecdh_ecdsa_3des_sha,ecdh_ecdsa_aes_128_sha,ecdh_ecdsa_aes_256_sha,ecdhe_ecdsa_null_sha,ecdhe_ecdsa_rc4_128_sha,ecdhe_ecdsa_3des_sha,ecdhe_ecdsa_aes_128_sha,ecdhe_ecdsa_aes_256_sha,ecdh_rsa_null_sha,ecdh_rsa_128_sha,ecdh_rsa_3des_sha,ecdh_rsa_aes_128_sha,ecdh_rsa_aes_256_sha,echde_rsa_null,ecdhe_rsa_rc4_128_sha,ecdhe_rsa_3des_sha,ecdhe_rsa_aes_128_sha,ecdhe_rsa_aes_256_sha,ecdhe_ecdsa_aes_128_gcm_sha_256,ecdhe_rsa_aes_128_gcm_sha_256
Hainish commented 8 years ago

I'm not interested in putting effort into supporting Red Hat, but if you want to open a PR to add support I'd be happy to review / merge, just @ me.

jeremyn commented 8 years ago

@Hainish The install-dev-dependencies.sh script optionally uses dnf to install packages (see here) which suggests that development is supported on Fedora. If Red Hat/Fedora is not officially supported, that should be noted somewhere that's easy to find.

Also, I submitted PR #5584 in June with some commits and documentation to improve Fedora support. Please feel free to review and comment there.

Hainish commented 8 years ago

I'll get a chance to look at #5584 next week

Hainish commented 8 years ago

@jeremyn now that #5584 is merged, your comment in https://github.com/EFForg/https-everywhere/issues/5628#issuecomment-235972197 seems like the appropriate fix - if you want to start working on providing a default ciphersuite for NSS that sgtm.

jeremyn commented 8 years ago

I created pull request #6215 to resolve this issue. Please feel free to close this issue after merging the pull request.