add crypto routines to stdlib

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

BPO	8998
Nosy	@malemburg, @loewis, @birkenfeld, @gpshead, @ncoghlan, @pitrou, @vstinner, @giampaolo, @tiran, @mcepl, @merwok, @davidmalcolm, @durban

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['type-feature', 'library'] title = 'add crypto routines to stdlib' updated_at = user = 'https://bugs.python.org/debatem1' ``` bugs.python.org fields: ```python activity = actor = 'gregory.p.smith' assignee = 'none' closed = True closed_date = closer = 'gregory.p.smith' components = ['Library (Lib)'] creation = creator = 'debatem1' dependencies = [] files = [] hgrepos = [] issue_num = 8998 keywords = [] message_count = 95.0 messages = ['107813', '107814', '107816', '107818', '107821', '107822', '107823', '107828', '107830', '107834', '107846', '107868', '107872', '107873', '107882', '107885', '108076', '108077', '108078', '108079', '108080', '108081', '108082', '108083', '108084', '108087', '108088', '108091', '108092', '108093', '108094', '108157', '108171', '108224', '108233', '108921', '108923', '108926', '108931', '108935', '115039', '115044', '115142', '116727', '116730', '116731', '116732', '116733', '116734', '116735', '116736', '116737', '116739', '116743', '116756', '116835', '116838', '116842', '116844', '116845', '116850', '116856', '116860', '116869', '116870', '116871', '116879', '116994', '116995', '116997', '117034', '117035', '117039', '117045', '117078', '117086', '117089', '117092', '117094', '117101', '117121', '117127', '118650', '118651', '118655', '118656', '118657', '118658', '118663', '120306', '194208', '194209', '194210', '194211', '194212'] nosy_count = 21.0 nosy_names = ['lemburg', 'loewis', 'georg.brandl', 'gregory.p.smith', 'exarkun', 'ncoghlan', 'pitrou', 'vstinner', 'giampaolo.rodola', 'christian.heimes', 'lorph', 'heikki', 'mcepl', 'eric.araujo', 'debatem1', 'dmalcolm', 'daniel.urban', 'mcrute', 'jsamuel', 'devin', 'madison.may'] pr_nums = [] priority = 'normal' resolution = 'later' stage = None status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue8998' versions = ['Python 3.4'] ```

[8726d1eb-a365-45b6-b81d-c75988975e5a]

I should note that I can't touch anything to do with Elliptic Curve crypto. I don't know if I can comment on the reasons for that.

Hopefully anything ECC related can be done separately. There's certainly no ECC APIs in Python now, so there would be no loss of functionality if any new crypto-related APIs didn't include ECC. Later if someone is sufficiently interesting, they could add it to the base library.

[pitrou]

Le vendredi 17 septembre 2010 à 23:14 +0000, Dave Malcolm a écrit :

Dave Malcolm \dmalcolm@redhat.com\ added the comment:

On Fri, 2010-09-17 at 23:11 +0000, Antoine Pitrou wrote: > Antoine Pitrou \pitrou@free.fr\ added the comment: > > > How about nss? As a bonus, this would also avoid making more work for > > Fedora (\http://fedoraproject.org/wiki/FedoraCryptoConsolidation\). > > Well, similar question: what will it bring and who will do the work? :) > (Fedora perhaps?)

Possibly me - if you'll take my patches :)

Converting/rewriting _ssl.c while remaining compatible is certainly the most interesting part, so I suggest you start with that if you are motivated :) You should also explain the rationale on the mailing-list, so that interested people have a chance to offer advice.

[davidmalcolm]

FWIW, one of my RH colleagues (John Dennis) has written a set of Python bindings for NSS: http://fedoraproject.org/wiki/Features/PythonNSS

(Though that seems to me to be a slightly different thing to a general-purpose crypto lib that happens to be written using NSS as an implementation detail)

[gpshead]

libtomcrypt is a _great_ library. That is what hashlib uses for the hash algorithms when OpenSSL is not available.

But the _primary_ reason for using OpenSSL is that it is the defacto open source location for the best architecture specific implementations of any hash and crypto algorithm. OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in most cases.

The NSS everywhere effort mentioned in the fedora link sounds interesting. I support having the ability to link against that instead of OpenSSL or copies of libtomcrypt but I am generally in favor of absolute performance per byte of all algorithms concerned being available. (ie: don't force hashlib to stop using openssl, just provide an alternative).

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

On Fri, Sep 17, 2010 at 8:55 PM, Gregory P. Smith \report@bugs.python.org\ wrote:

Gregory P. Smith \greg@krypto.org\ added the comment:

libtomcrypt is a _great_ library.  That is what hashlib uses for the hash algorithms when OpenSSL is not available.

But the _primary_ reason for using OpenSSL is that it is the defacto open source location for the best architecture specific implementations of any hash and crypto algorithm.  OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in most cases.

The NSS everywhere effort mentioned in the fedora link sounds interesting.  I support having the ability to link against that instead of OpenSSL or copies of libtomcrypt but I am generally in favor of absolute performance per byte of all algorithms concerned being available.  (ie: don't force hashlib to stop using openssl, just provide an alternative).

I'm open to working with other libraries, but realistically there isn't a huge point in supporting every crypto library out there, and it would be a bad idea all around to try. My suggestion would be that we stick with OpenSSL until a replacement for _ssl.c exists; after that we can revisit that decision and see where we stand.

Geremy Condra

[b993299a-f990-4540-b708-2c574aa9e4db]

OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in most cases.

Gregory, do you have any evidence to substantiate this claim? Not that it isn't plausible, but I couldn't find any benchmarks, and here the author of libtomcrypt finds it to be 40% faster than OpenSSL concerning RSA operations.

http://www.adras.com/TomsFastMath-faster.t71-93.html

but I am generally in favor of absolute performance per byte of all algorithms concerned being available

Performance isn't all that matters, or else Python would have used GMP, as Guido discussed here:

http://mail.python.org/pipermail/python-3000/2007-September/010329.html

It is also not a convincing argument that new python libraries should use OpenSSL if possible just because that is what _ssl uses. Compiling Python with OpenSSL support has been optional because it puts additional restrictions on the PSF license. Spreading this restriction to the future crypto module (when we have a choice not to) doesn't make sense.

[pitrou]

It is also not a convincing argument that new python libraries should use OpenSSL if possible just because that is what _ssl uses. Compiling Python with OpenSSL support has been optional because it puts additional restrictions on the PSF license. Spreading this restriction to the future crypto module (when we have a choice not to) doesn't make sense.

It certainly makes more sense than making Python depend on *several* crypto libraries. As for the licensing restriction, it doesn't seem to disturb many Python users. It's the first time I see someone complaining about it.

This isn't meant to discourage any such efforts (I don't care about which crypto library we use), but any proposal of using another crypto library than OpenSSL should IMO include a migration proposal for the _ssl module.

[b993299a-f990-4540-b708-2c574aa9e4db]

It certainly makes more sense than making Python depend on *several* crypto libraries.

Since libtomcrypt is public domain, you could incorporate the source into the tree without making it a binary dependency. The same cannot be said for OpenSSL. I certainly wouldn't mind having 1 dependency on NSS, but having 2 modules depend on OpenSSL is a step in the wrong direction.

As for the licensing restriction, it doesn't seem to disturb many Python users. It's the first time I see someone complaining about it.

It took several years until someone like Marc-Andre Lemburg to find that the Python website might be violating that license. Perhaps the reason is because no one bothers to read licenses carefully. People are probably violating the license without knowing it.

If you take a look at the clause "All advertising materials mentioning features or use of this software must display the following acknowledgment", you will find at least 2 problems.

One is that if you mention something like "base64" in whatever could be deemed "advertising", you will be subject to this clause because base64 is a feature of OpenSSL, even if you don't use their implementation. Another problem is the difference between the clause "features or use of this software", which is semantically different from "features of this software or use of this software".

Is it worth the risk to depend on Eric Young's proclivity to sue now that he works for RSA and produces competing software called BSAFE? Maybe it is for you, but certainly not for me.

[pitrou]

Since libtomcrypt is public domain, you could incorporate the source into the tree without making it a binary dependency.

And then we have to maintain our copy ourselves. I'm not sure why you think this is better than depending on a system-wide install, because it's certainly worse.

(we do have private copies of a couple of libraries: zlib, expat, libffi. The first two are probably for historical reasons (the system-wide versions are used by default), while the third is because it's patched)

I certainly wouldn't mind having 1 dependency on NSS, but having 2 modules depend on OpenSSL is a step in the wrong direction.

Perhaps you wouldn't mind, but others would (especially packagers; including ourselves since we build binary packages for Windows and Mac OS X).

It took several years until someone like Marc-Andre Lemburg to find that the Python website might be violating that license. Perhaps the reason is because no one bothers to read licenses carefully. People are probably violating the license without knowing it.

The solution to stop violating it is trivial, though: just add the required mention(s). Compare that to rewriting a lot of code and making sure it doesn't change behaviour compared to previous Python versions.

One is that if you mention something like "base64" in whatever could be deemed "advertising", you will be subject to this clause because base64 is a feature of OpenSSL, even if you don't use their implementation.

Unless "base64" is an OpenSSL trademark, this is FUD.

[b993299a-f990-4540-b708-2c574aa9e4db]

The solution to stop violating it is trivial, though: just add the required mention(s).

That only solves the problem for Python.org. It does not solve the problem for everyone else that has to write "This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/) This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)" in advertisement that mentions features.

Unless "base64" is an OpenSSL trademark, this is FUD.

The license clearly states: "All advertising materials mentioning features or use of this software". Do you somehow disagree that base64 is a feature of the OpenSSL library?

http://www.openssl.org/docs/crypto/BIO_f_base64.html

[gpshead]

This bug has turned into a bikeshed.

Lets stop that please.

I _DON'T_ care about performance when it comes to someone submitting an actual working implementation of a crypto library for inclusion with the standard library. The first priority needs to be a useful stable API.

After that is settled, implementations backed by whatever libraries can be written by anyone who wants to contribute them. Lets not stop that from happening by arguing about who's library is a prettier color. The first one may well be implemented in Python itself as a for all I care.

-gps

(comments below are to answer questions but really are not relevant to the bug)

On Sat, Sep 18, 2010 at 3:54 PM, lorph \report@bugs.python.org\ wrote:

lorph \lorph1@gmail.com\ added the comment:

> OpenSSL outperforms libtomcrypt by a significant factor (easily 2x) in most cases.

Gregory, do you have any evidence to substantiate this claim? Not that it isn't plausible, but I couldn't find any benchmarks, and here the author of libtomcrypt finds it to be 40% faster than OpenSSL concerning RSA operations.

I should not have said "most" cases but there are many cases where it does. Run your own microbenchmarks on various hardware if you're curious ("openssl speed algorithmname" on the command line is an existing openssl benchmark). The important thing to realize is that libtomcrypt is intentionally written in very portable C. That is great but it leaves a lot on the table. Optimizations for various platforms to take advantage of enhanced instruction sets such as SSE2 and explicit hardware crypto acceleration instructions such as http://software.intel.com/en-us/articles/intel-advanced-encryption-standard-aes-instructions-set/are not likely to be part of libtomcrypt, nor should they ever be part of code included with and distributed as part of Python. That should come from a dynamic library that is part of the OS. That may be NSS, that may be OpenSSL, that may be any number of other things. Its beyond the scope of this bug.

I'm sorry for bringing the performance angle up even if it motivate me to make hashlib happen.

An example benchmark of hashlib using libtomcrypt sha1 vs openssl sha1 is here http://bugs.python.org/issue1121611#msg47779 if you're curious but please make any responses to that outside of this bug.

-gps

[birkenfeld]

The license clearly states: "All advertising materials mentioning features or use of this software". Do you somehow disagree that base64 is a feature of the OpenSSL library?

http://www.openssl.org/docs/crypto/BIO_f_base64.html

That's funny. Do you think that if OpenSSL provided its own implementation of strlen(), every text that mentions strlen() needs to acknowledge OpenSSL? Do you realize how ridiculous that is?

[malemburg]

Just another data point for the discussion:

The PSF is currently funding the effort to port pyOpenSSL to Python 3.x and the port is nearly finished.

It may be worthwhile investigating adding the EVP interface from evpy (with the ctypes bindings converted to real C wrappers) to pyOpenSSL and then adding the pyOpenSSL package to the stdlib.

Note that going for other crypto libs than OpenSSL is currently not an option, since those are not widely available on the OSes and we cannot easily add crypto code itself to Python's stdlib due to the issues with crypto import/export/use restrictions which would limit the use of Python in various countries.

[pitrou]

It may be worthwhile investigating adding the EVP interface from evpy (with the ctypes bindings converted to real C wrappers) to pyOpenSSL and then adding the pyOpenSSL package to the stdlib.

pyOpenSSL being LGPL'ed, I'm not sure this is possible. On the other hand, gradually adding some of pyOpenSSL's most useful functionalities to the ssl module would be worthwhile, and I know Jean-Paul would be interested in this. This has already begun in 3.2, but I've been alone in doing it and it would be nice if other people contributed: http://docs.python.org/dev/library/ssl.html#ssl-contexts

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

> It may be worthwhile investigating adding the EVP interface > from evpy (with the ctypes bindings converted to real C wrappers) > to pyOpenSSL and then adding the pyOpenSSL package to the stdlib.

pyOpenSSL being LGPL'ed, I'm not sure this is possible.

Changing the license should be possible, since we know the copyright owners and both are PSF members (AB Strakt and Jean-Paul).

On the other hand, gradually adding some of pyOpenSSL's most useful functionalities to the ssl module would be worthwhile, and I know Jean-Paul would be interested in this. This has already begun in 3.2, but I've been alone in doing it and it would be nice if other people contributed: http://docs.python.org/dev/library/ssl.html#ssl-contexts

I don't think we should redo this effort in the context of the ssl module.

pyOpenSSL is stable, in production use and has a decent API. The ssl module is good enough for HTTPS client use. pyOpenSSL provides a robust server side implementation with all the required certificate and context handling needed for this.

We could tell people to use the ssl module for clients and pyOpenSSL for the server side and perhaps integrate the OpenSSL package into the ssl namespace.

[pitrou]

pyOpenSSL is stable, in production use and has a decent API. The ssl module is good enough for HTTPS client use. pyOpenSSL provides a robust server side implementation with all the required certificate and context handling needed for this.

We could tell people to use the ssl module for clients and pyOpenSSL for the server side and perhaps integrate the OpenSSL package into the ssl namespace.

In this case, this should be decided early, so that I know if I should continue caring about the ssl module or not. I'm not interested in maintaining potentially obsolete code.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

> Unless "base64" is an OpenSSL trademark, this is FUD.

The license clearly states: "All advertising materials mentioning features or use of this software". Do you somehow disagree that base64 is a feature of the OpenSSL library?

What specific "advertising material" mentions "base64" but fails to mention "OpenSSL"? In any case, this is off-topic for this issue.

[b993299a-f990-4540-b708-2c574aa9e4db]

Do you think that if OpenSSL provided its own implementation of strlen(), every text that mentions strlen() needs to acknowledge OpenSSL? Do you realize how ridiculous that is?

If that text is deemed to be advertising by Eric Young and a court of law, then absolutely yes. The license is short, clear, and does not make any exceptions for features that you might deem to be commonplace.

http://www.openssl.org/source/license.html

I also agree that this is ridiculous, but believing something is ridiculous does not make it any less real as Barnes and Nobles learned the hard way by using Amazon's one-click patent.

The important thing to realize is that libtomcrypt is intentionally written in very portable C. That is great but it leaves a lot on the table. Optimizations for various platforms to take advantage of enhanced instruction sets such as SSE2 and explicit hardware crypto acceleration instructions such as [...] are not likely to be part of libtomcrypt,

A quick glance at libtomcrypt tells me otherwise. It is portable C, but it still has inline assembler. It has optimizations using SSE2, x86, x86_64, and PPC32. Even though it might not have that new Intel AES instruction yet, this is the same argument people had for using GMP for python's integers.

If there is a problem with multiple libraries, I'd like to reiterate my support for migrating to NSS.

[pitrou]

If there is a problem with multiple libraries, I'd like to reiterate my support for migrating to NSS.

Will your support go so far as to investigate feasibility and provide a patch?

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

On Mon, Sep 20, 2010 at 3:44 PM, lorph \report@bugs.python.org\ wrote:

lorph \lorph1@gmail.com\ added the comment:

> Do you think that if OpenSSL provided its own implementation of strlen(), every text that mentions strlen() needs to acknowledge OpenSSL? Do you realize how ridiculous that is?

If that text is deemed to be advertising by Eric Young and a court of law, then absolutely yes. The license is short, clear, and does not make any exceptions for features that you might deem to be commonplace.

http://www.openssl.org/source/license.html

I also agree that this is ridiculous, but believing something is ridiculous does not make it any less real as Barnes and Nobles learned the hard way by using Amazon's one-click patent.

I get called paranoid a lot, but I don't find this terribly convincing. Having said that, I'm happy to have your help in making sure that the interface we create is agnostic with respect to openssl, libtomcrypt, NSS, etc. That would mean that if Eric Young did decide to try and force his hand we would be able to switch away the very next day with no muss and no fuss. Are you willing to provide that help?

> The important thing to realize is that libtomcrypt is intentionally written in very portable C.  That is great but it leaves a lot on the table. Optimizations for various platforms to take advantage of enhanced instruction sets such as SSE2 and explicit hardware crypto acceleration instructions such as [...] are not likely to be part of libtomcrypt,

A quick glance at libtomcrypt tells me otherwise. It is portable C, but it still has inline assembler. It has optimizations using SSE2, x86, x86_64, and PPC32. Even though it might not have that new Intel AES instruction yet, this is the same argument people had for using GMP for python's integers.

I'm not worried about speed. In fact, if fast means "vulnerable to side-channel cryptanalysis" I'm firmly opposed to it, and I don't know if that's the case here. OpenSSL has at least been subject to significant attention in that regard.

If there is a problem with multiple libraries, I'd like to reiterate my support for migrating to NSS.

I'm happy to do this work if you (or someone else) will step forward and help me identify and fix compatibility concerns. Otherwise, this is just a waste of time all around.

Geremy Condra

[birkenfeld]

If that text is deemed to be advertising by Eric Young and a court of law

The license of a software product cannot affect software that is not even aware of that said product. (A patent, or a trademark can.) It governs the use of that product, not of others. "Features of this library" means really features of *that* library, as specifically implemented in that library.

If a license for one specific software product could affect all other software with such a primitive clause, I don't want to think how Richard Stallman would have worded the GPL :)

[b993299a-f990-4540-b708-2c574aa9e4db]

The license of a software product cannot affect software that is not even aware of that said product.

I never claimed that the clause triggered for all software in existence. We are talking about OpenSSL being bundled with Python where Python is very much aware of OpenSSL. Provided the following 3 circumstances are met, the advertisement clause applies:

1. You are distributing Python with OpenSSL
2. You are "advertising".
3. Your advertising mentions features.

By mentioning features of Python, or even a feature of OpenSSL you don't even use (base64), by the wording of the license you are obligated to also advertise OpenSSL and Eric Young. This obviously has a chilling effect on the distribution and advertising of Python apps. Think about the 100 char blurb on every small web banner.

In fact, if fast means "vulnerable to side-channel cryptanalysis" I'm firmly opposed to it, and I don't know if that's the case here. OpenSSL has at least been subject to significant attention in that regard.

LTC does address side-channel attacks, but this is a moot point since by using a high level language like Python, you are vulnerable to memory scanning since you cannot normally zero out Python strings (something you may wish to consider in the crypto API).

I'd also add that the "rounds" option should be left in for compatibility reasons. For easy usage, a default such as CBC could be specified. Otherwise, I don't think there is anything wrong with the API.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

I never claimed that the clause triggered for all software in existence. We are talking about OpenSSL being bundled with Python where Python is very much aware of OpenSSL. Provided the following 3 circumstances are met, the advertisement clause applies:

Can we please stop getting license interpretations from laymen on a bug tracker? If somebody is really concerned that the PSF might violate some license, the PSF lawyer should be asked to evaluate the situation.

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

> pyOpenSSL is stable, in production use and > has a decent API. The ssl module is good enough for HTTPS client > use. pyOpenSSL provides a robust server side implementation with > all the required certificate and context handling needed for this. > > We could tell people to use the ssl module for clients and > pyOpenSSL for the server side and perhaps integrate the OpenSSL > package into the ssl namespace.

In this case, this should be decided early, so that I know if I should continue caring about the ssl module or not. I'm not interested in maintaining potentially obsolete code.

I'll ask Jean-Paul and AB Strakt if they are up to contributing the pyOpenSSL code to the Python stdlib based on a contributor agreement. This would enable us to relicense the code undert the PSF license even if the original code's license is not changed.

Once that's a done deal, we can then consider moving in the above direction.

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

On Tue, Sep 21, 2010 at 4:04 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> Antoine Pitrou wrote:
>>
>> Antoine Pitrou <pitrou@free.fr> added the comment:
>>
>>> pyOpenSSL is stable, in production use and
>>> has a decent API. The ssl module is good enough for HTTPS client
>>> use. pyOpenSSL provides a robust server side implementation with
>>> all the required certificate and context handling needed for this.
>>>
>>> We could tell people to use the ssl module for clients and
>>> pyOpenSSL for the server side and perhaps integrate the OpenSSL
>>> package into the ssl namespace.
>>
>> In this case, this should be decided early, so that I know if I should
>> continue caring about the ssl module or not. I'm not interested in
>> maintaining potentially obsolete code.
>
> I'll ask Jean-Paul and AB Strakt if they are up to contributing
> the pyOpenSSL code to the Python stdlib based on a contributor
> agreement. This would enable us to relicense the code undert
> the PSF license even if the original code's license is not
> changed.
>
> Once that's a done deal, we can then consider moving in the above
> direction.

I'm not sure I understand the relevance of pyopenssl here- it's pretty clearly focused on SSL/TLS rather than on crypto. Maybe someone can clarify?

Geremy Condra

[malemburg]

geremy condra wrote:
> 
> geremy condra <debatem1@gmail.com> added the comment:
> 
> On Tue, Sep 21, 2010 at 4:04 AM, Marc-Andre Lemburg
> <report@bugs.python.org> wrote:
>>
>> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>>
>> Antoine Pitrou wrote:
>>>
>>> Antoine Pitrou <pitrou@free.fr> added the comment:
>>>
>>>> pyOpenSSL is stable, in production use and
>>>> has a decent API. The ssl module is good enough for HTTPS client
>>>> use. pyOpenSSL provides a robust server side implementation with
>>>> all the required certificate and context handling needed for this.
>>>>
>>>> We could tell people to use the ssl module for clients and
>>>> pyOpenSSL for the server side and perhaps integrate the OpenSSL
>>>> package into the ssl namespace.
>>>
>>> In this case, this should be decided early, so that I know if I should
>>> continue caring about the ssl module or not. I'm not interested in
>>> maintaining potentially obsolete code.
>>
>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>> the pyOpenSSL code to the Python stdlib based on a contributor
>> agreement. This would enable us to relicense the code under
>> the PSF license even if the original code's license is not
>> changed.
>>
>> Once that's a done deal, we can then consider moving in the above
>> direction.
> 
> I'm not sure I understand the relevance of pyopenssl here- it's pretty
> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
> clarify?

Yes, but it provides a decent platform for adding other crypto APIs as well and then we could have these as C APIs rather than wrappers using ctypes.

There's already a patch available from Keyphrene adding all those bits: http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en

The patch would need to be updated for the new pyOpenSSL version, but that's certainly within range. And we'd need to get their permission to relicense as well.

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

On Tue, Sep 21, 2010 at 10:33 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> geremy condra wrote:
>>
>> geremy condra <debatem1@gmail.com> added the comment:
>>
>> On Tue, Sep 21, 2010 at 4:04 AM, Marc-Andre Lemburg
>> <report@bugs.python.org> wrote:
>>>
>>> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>>>
>>> Antoine Pitrou wrote:
>>>>
>>>> Antoine Pitrou <pitrou@free.fr> added the comment:
>>>>
>>>>> pyOpenSSL is stable, in production use and
>>>>> has a decent API. The ssl module is good enough for HTTPS client
>>>>> use. pyOpenSSL provides a robust server side implementation with
>>>>> all the required certificate and context handling needed for this.
>>>>>
>>>>> We could tell people to use the ssl module for clients and
>>>>> pyOpenSSL for the server side and perhaps integrate the OpenSSL
>>>>> package into the ssl namespace.
>>>>
>>>> In this case, this should be decided early, so that I know if I should
>>>> continue caring about the ssl module or not. I'm not interested in
>>>> maintaining potentially obsolete code.
>>>
>>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>>> the pyOpenSSL code to the Python stdlib based on a contributor
>>> agreement. This would enable us to relicense the code under
>>> the PSF license even if the original code's license is not
>>> changed.
>>>
>>> Once that's a done deal, we can then consider moving in the above
>>> direction.
>>
>> I'm not sure I understand the relevance of pyopenssl here- it's pretty
>> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
>> clarify?
>
> Yes, but it provides a decent platform for adding other crypto APIs
> as well and then we could have these as C APIs rather than wrappers
> using ctypes.

The intention all along has been that we use the C API, and in fact I'm pretty far along on writing that. Assuming there won't be an issue with porting pyopenssl to python3, this seems like a pretty good idea to me though.

There's already a patch available from Keyphrene adding all those bits: http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en

The patch would need to be updated for the new pyOpenSSL version, but that's certainly within range. And we'd need to get their permission to relicense as well.

Bits and pieces of this look useful but it also looks like I'd be rewriting a lot of it to move away from the RSA_* routines, etc. I suspect it would take more time to get it into line than to just cherrypick out of it.

Geremy Condra

[malemburg]

geremy condra wrote:
> 
>>>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>>>> the pyOpenSSL code to the Python stdlib based on a contributor
>>>> agreement. This would enable us to relicense the code under
>>>> the PSF license even if the original code's license is not
>>>> changed.
>>>>
>>>> Once that's a done deal, we can then consider moving in the above
>>>> direction.
>>>
>>> I'm not sure I understand the relevance of pyopenssl here- it's pretty
>>> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
>>> clarify?
>>
>> Yes, but it provides a decent platform for adding other crypto APIs
>> as well and then we could have these as C APIs rather than wrappers
>> using ctypes.
> 
> The intention all along has been that we use the C API, and in fact
> I'm pretty far along on writing that. Assuming there won't be an issue
> with porting pyopenssl to python3, this seems like a pretty good idea
> to me though.

Ah, sorry, I must have missed that turn in the discussion :-)

The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul is planning for an alpha release next month.

> There's already a patch available from Keyphrene adding all those bits: > http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en > > The patch would need to be updated for the new pyOpenSSL version, > but that's certainly within range. And we'd need to get their permission > to relicense as well.

Bits and pieces of this look useful but it also looks like I'd be rewriting a lot of it to move away from the RSA_* routines, etc. I suspect it would take more time to get it into line than to just cherrypick out of it.

If you are writing new code for this anyway, it may be better to avoid the license question of the Keyphrene patch and just use their code as reference.

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

On Tue, Sep 21, 2010 at 11:29 AM, Marc-Andre Lemburg
<report@bugs.python.org> wrote:
>
> Marc-Andre Lemburg <mal@egenix.com> added the comment:
>
> geremy condra wrote:
>>
>>>>> I'll ask Jean-Paul and AB Strakt if they are up to contributing
>>>>> the pyOpenSSL code to the Python stdlib based on a contributor
>>>>> agreement. This would enable us to relicense the code under
>>>>> the PSF license even if the original code's license is not
>>>>> changed.
>>>>>
>>>>> Once that's a done deal, we can then consider moving in the above
>>>>> direction.
>>>>
>>>> I'm not sure I understand the relevance of pyopenssl here- it's pretty
>>>> clearly focused on SSL/TLS rather than on crypto. Maybe someone can
>>>> clarify?
>>>
>>> Yes, but it provides a decent platform for adding other crypto APIs
>>> as well and then we could have these as C APIs rather than wrappers
>>> using ctypes.
>>
>> The intention all along has been that we use the C API, and in fact
>> I'm pretty far along on writing that. Assuming there won't be an issue
>> with porting pyopenssl to python3, this seems like a pretty good idea
>> to me though.
>
> Ah, sorry, I must have missed that turn in the discussion :-)
>
> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
> is planning for an alpha release next month.

Do you know if he's looking for help with that? There's been some talk of a porting sprint here and I'd be happy to put that at the top of the list.

>> There's already a patch available from Keyphrene adding all those bits: >> http://www.keyphrene.com/products/pyOpenSSL-extended/index.php?lng=en >> >> The patch would need to be updated for the new pyOpenSSL version, >> but that's certainly within range. And we'd need to get their permission >> to relicense as well. > > Bits and pieces of this look useful but it also looks like I'd be > rewriting a lot of it to move away from the RSA_* routines, etc. I > suspect it would take more time to get it into line than to just > cherrypick out of it.

If you are writing new code for this anyway, it may be better to avoid the license question of the Keyphrene patch and just use their code as reference.

Yeah, I think that makes the most sense.

Geremy Condra

[malemburg]

geremy condra wrote:
> 
>>> The intention all along has been that we use the C API, and in fact
>>> I'm pretty far along on writing that. Assuming there won't be an issue
>>> with porting pyopenssl to python3, this seems like a pretty good idea
>>> to me though.
>>
>> Ah, sorry, I must have missed that turn in the discussion :-)
>>
>> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
>> is planning for an alpha release next month.
> 
> Do you know if he's looking for help with that? There's been some talk of
> a porting sprint here and I'd be happy to put that at the top of the list.

I don't know. You might want to contact him directly.

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

>> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul >> is planning for an alpha release next month. > > Do you know if he's looking for help with that? There's been some talk of > a porting sprint here and I'd be happy to put that at the top of the list.

I don't know. You might want to contact him directly.

Sent.

By the way, if you've used the keyphrene API and wouldn't mind sharing how it compares with the proposed crypto API I'd appreciate it, I don't think it came up in any of the initial conversations about this.

Geremy Condra

[malemburg]

geremy condra wrote:
> 
> geremy condra <debatem1@gmail.com> added the comment:
> 
>>>> The pyOpenSSL port to Python3 is closing in on completion. Jean-Paul
>>>> is planning for an alpha release next month.
>>>
>>> Do you know if he's looking for help with that? There's been some talk of
>>> a porting sprint here and I'd be happy to put that at the top of the list.
>>
>> I don't know. You might want to contact him directly.
> 
> Sent.
> 
> By the way, if you've used the keyphrene API and wouldn't mind sharing
> how it compares with the proposed crypto API I'd appreciate it, I
> don't think it came up in any of the initial conversations about this.

No, I haven't used the APIs. I do have some experience with mxCrypto, though, which is a low-level wrapper for the ciphers and hashes in OpenSSL:

http://www.egenix.com/www2002/python/mxCrypto.html

[malemburg]

I've been in touch with the copyright holders of pyOpenSSL and they all were positive about contributing the code to the PSF under a contributor agreement.

The idea would then be to add the crypto routines to pyOpenSSL and have that added to the stdlib as say openssl package.

So how should we go about this ? Open a new ticket ?

[pitrou]

I've been in touch with the copyright holders of pyOpenSSL and they all were positive about contributing the code to the PSF under a contributor agreement. So how should we go about this ? Open a new ticket ?

I would like to see public discussion about this, especially in the light of the ssl module improvements in Python 3.2. It is not obvious that duplicate APIs in the stdlib are a good idea, especially when they are not compatible with each other. It also means that the current pyOpenSSL maintainer (Jean-Paul) should agree to do maintenance directly in the stdlib rather than in a separate repo.

The idea would then be to add the crypto routines to pyOpenSSL and have that added to the stdlib as say openssl package.

This sounds a bit ridiculous. Why not add the crypto routines directly to the stdlib?

[malemburg]

Antoine Pitrou wrote:

Antoine Pitrou \pitrou@free.fr\ added the comment:

> I've been in touch with the copyright holders of pyOpenSSL and they > all were positive about contributing the code to the PSF under a > contributor agreement. > So how should we go about this ? Open a new ticket ?

I would like to see public discussion about this, especially in the light of the ssl module improvements in Python 3.2. It is not obvious that duplicate APIs in the stdlib are a good idea, especially when they are not compatible with each other. It also means that the current pyOpenSSL maintainer (Jean-Paul) should agree to do maintenance directly in the stdlib rather than in a separate repo.

> The idea would then be to add the crypto routines to pyOpenSSL and > have that added to the stdlib as say openssl package.

This sounds a bit ridiculous. Why not add the crypto routines directly to the stdlib?

To make those routines available to a broader audience and to get more user feedback.

I don't think we can add pyOpenSSL to Python 3.2, so it's better to use the available time to hash out the details outside the stdlib. Once it's in the stdlib, changing APIs is very difficult.

[pitrou]

> This sounds a bit ridiculous. Why not add the crypto routines directly > to the stdlib?

To make those routines available to a broader audience and to get more user feedback.

Sure. But it can be any standalone package, not necessarily pyOpenSSL. Then, if we want to add them to the stdlib, we don't have to pull in the whole pyOpenSSL package.

I don't think we can add pyOpenSSL to Python 3.2,

Right, it's too late.

so it's better to use the available time to hash out the details outside the stdlib. Once it's in the stdlib, changing APIs is very difficult.

Then I think the discussion about API and process should move to python-ideas.

[malemburg]

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>>> This sounds a bit ridiculous. Why not add the crypto routines directly
>>> to the stdlib?
>>
>> To make those routines available to a broader audience and to
>> get more user feedback.
> 
> Sure. But it can be any standalone package, not necessarily pyOpenSSL.
> Then, if we want to add them to the stdlib, we don't have to pull in the
> whole pyOpenSSL package.

pyOpenSSL has the advantage of already providing all the other bits and pieces needed to interface and build against OpenSSL, so it's a good ecosystem for such a development.

Besides there are already patches available which do add the ciphers and hashs to pyOpenSSL, so the development could be sped up by using those as references.

> I don't think we can add pyOpenSSL to Python 3.2,

Right, it's too late.

> so it's better > to use the available time to hash out the details outside the > stdlib. Once it's in the stdlib, changing APIs is very difficult.

Then I think the discussion about API and process should move to python-ideas.

The APIs should probably be discussed on the Python crypto or pyOpenSSL list and the discussion about its integration into the stdlib on either the python-dev or the stdlib list.

https://lists.sourceforge.net/lists/listinfo/pyopenssl-list http://mail.python.org/mailman/listinfo/python-crypto http://mail.python.org/mailman/listinfo/python-dev http://mail.python.org/mailman/listinfo/stdlib-sig

python-ideas is not really meant for such discussions.

[pitrou]

The APIs should probably be discussed on the Python crypto or pyOpenSSL list and the discussion about its integration into the stdlib on either the python-dev or the stdlib list.

If the target goal is stdlib inclusion, the APIs should be discussed on python-ideas too.

python-ideas is not really meant for such discussions.

Yes, it is.

[e3cd80c0-0d1c-412a-b4b4-5e542ba3ba75]

Besides there are already patches available which do add the ciphers and hashs to pyOpenSSL, so the development could be sped up by using those as references.

I don't think that's the case. I admit that development on this has been very slow (curse you, Real Life!) but it is nearing a release, and I doubt that this would save any time even if I didn't have to rewrite half of the keyphrene code.

Geremy Condra

[2fb7f418-2576-4ac8-a7c4-99ab5ffbee45]

It sounds like you may already have an idea of how you want the API structured, but just in case you're still thinking about it here's another API to look at that I think focuses on exactly what you were highlighting as priorities (sane defaults, easy to use): keyczar.

I use keyczar quite a bit and really like it, although I tend to only use the key classes directly instead of using the generic "Crypter" etc. classes.

http://code.google.com/p/keyczar/

[bed36d65-e285-4d18-acf3-5dd5d6a271d3]

This issue may have been dead for 3+ years, but perhaps it's time its brought back to the surface. Aside from simple being convenient for general security practices, a stdlib module for crypto routines would enable python to handle encrypted zipfiles and resolve issues like bpo-9170. Currently, only hashlib and hmac are available to users (see http://docs.python.org/3/library/crypto.html).

I'd imagine that collaboration with the likes of Dwayne from PyCrypto or further collaboration with Geremy from Evpy would be possible, and perhaps the stdlib could build on a 3rd party crypto library. Is anyone else interested in working to make this happen (or does anyone have a good argument against a stdlib crypto library)? I imagine it would be difficult to maintain, but perhaps its at least worth giving this issue a second chance.

[malemburg]

Madison May wrote:

This issue may have been dead for 3+ years, but perhaps it's time its brought back to the surface. Aside from simple being convenient for general security practices, a stdlib module for crypto routines would enable python to handle encrypted zipfiles and resolve issues like bpo-9170. Currently, only hashlib and hmac are available to users (see http://docs.python.org/3/library/crypto.html).

I'd imagine that collaboration with the likes of Dwayne from PyCrypto or further collaboration with Geremy from Evpy would be possible, and perhaps the stdlib could build on a 3rd party crypto library. Is anyone else interested in working to make this happen (or does anyone have a good argument against a stdlib crypto library)? I imagine it would be difficult to maintain, but perhaps its at least worth giving this issue a second chance.

IMO, it's better to have crypto routines not be part of the stdlib. Not because there's a technical reason. Making crypto code part of the stdlib would create legal issues, e.g. make it illegal to use in some parts of the world.

See http://www.cryptolaw.org/ for details.

[bed36d65-e285-4d18-acf3-5dd5d6a271d3]

Yeah, that definitely qualifies as a good argument. I didn't consider the legal issues that would create. Let's let this issue rest in peace, then.

[gpshead]

The interest is there but nobody is designing or implementing it. Realistically, make a module sporting the proposed API and put it up on pypi and after success there: raise the issue on python-ideas.

Read the entire thread here first before designing such a thing.

We will always have unsolvable problems in any crypto shipped with Python such as side channel attacks. Those are outside the scope of what the CPython project should ever attempt to solve. Document them as weaknesses and suggest people use C libraries dedicated to avoiding those problems if their security needs require it.

[gpshead]

Python already ships with SSL support so the legal ship might have sailed ages ago if thats what PSF lawyers say. (IANAL)