Evaluate removing GnuPG in place of a pure Python PGP implementation

[fpietrosanti]

This ticket is to evaluate removing GnuPG in place of a pure Python PGP implementation.

Currently GnuPG is an heavy dependency that we just use for:

• PGP Encryption of files
• PGP Encryption of emails
• PGP Key Validation (before Import)

If we would be able to migrate easily to https://github.com/singpolyma/OpenPGP-Python we would be able to remove the fat Python-GnuPG dependency.

We would also be able to simplify the AppArmor profile by removing /bin/sh allowance.

Because we switched to "cryptography" removing PyCrypto we cannot use it, but we would need to integrate https://github.com/singpolyma/OpenPGP-Python/issues/1 and then evaluate what we are missing.

We should additionally evaluate if that library operate in "streaming" or if it does data buffering.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[fpietrosanti]

/cc @vecna

[fpietrosanti]

@singpolyma may provide some hints

[singpolyma]

As to streaming, the current implementation uses bytestrings, and I don't have a streaming decryption mode, but that should be doable.

[singpolyma]

What would be your preferred API for streaming stuff?

[evilaliv3]

currently we simply need to do encryption so no problem if you are still missing a streaming function for decryption.

probably it would be interesting to migrate also OpenPGP-Python to cryptography library that nowadays is more maintained. this will allow more easy (and small) integration with pypenssl and a lot of other software that are doing the same migration.

@singpolyma what do you think? (thank you for your fast reply)

[fpietrosanti]

@evilaliv3 The migration to cryptography sounds like to be done https://github.com/singpolyma/OpenPGP-Python/issues/1 .

@singpolyma It's just highly relevant for GlobaLeaks software to be "lightweight", so to be able to process a 10GB file on a small Linux VPS with 128MB of RAM. For that reason it's important that the "data processing" does not make buffering for the entire size of the files that we are going to process. That was my "meaning" for streaming

[singpolyma]

I would say that while cryptography might be more maintained, it's also scary-new. That said, the OpenPGP library is designed to make choosing different backends possible, so I'm definitely going to add a backend for cryptgography -- have already started on it.

@fpietrosanti right, I know what streaming is, I meant what would you want the API to look like for that? Or, perhaps, what's the most idiomatic way to do it? Should I make my parser re-feedable or take a generator or what?

[evilaliv3]

@singpolyma :

i've no particular suggestion for the API;

from our point of view i expect you will implement a file like api with something like this for streaming encryption:

• open(pgpkey, file, 'w')
• write(chunk)
• close()

what do you think?

[singpolyma]

I will do more research on how streaming is normally handled in python for the cleanest API :)

[fpietrosanti]

@singpolyma It would be amazing to make something that's similar or compatible to the Python-GnuPG API, in order to provide an easy replacement path to anyone that's using the GnuPG based wrapper. But that's more a desiderata

[singpolyma]

Update on some of this -- I'm about half done adding support for cryptography, but cryptography lacks too many features to complete this right now (they don't even have RSA encrypt/decrypt support yet) -- will revisit that part of it when the upstream is more mature

[public]

@singpolyma We should be landing RSA encrypt in 0.4. Are there any other features in particular you are after?

[singpolyma]

@public DSA sign/verify support would be very high on my list, but I can at least get a version of OpenPGP (with RSA) working without it. I will revisit this with the 0.4 release, then

[public]

Hopefully DSA will be in 0.4 too :)

[fpietrosanti]

@public @singpolyma I love you guys, and we'd love to be the very first user of OpenPGP-Python+Cryptography.

Once the full OpenPGP-Python support is in place, during the integration into GlobaLeaks we could consider to bring OpenPGP-Python to became a Twisted (https://pypi.python.org/pypi/Twisted/13.2.0) module, so that also other projects that may need it could use it? /cc @zooko

[singpolyma]

I haven't forgotten about you guys, but have had a rash of "actual work" to do -- hopefully I'll get back to this soon :)

[evilaliv3]

eheheh thanks @singpolyma

[origliante]

@fpietrosanti will be outdated

[fpietrosanti]

@origliante why is outdated? PGP notification will still be used by notifications

[evilaliv3]

from latest developments i would suggest to switch in direction of PGPy, https://github.com/globaleaks/GlobaLeaks/issues/1229

[hellais]

I am a bit skeptical of switching over to a custom non audited PGP implementation. From the brief look at the code done some time ago ti seems solid, but it would be ideal to have some cryptographers audit it before doing the switch.

I would move this onto a later milestone than April 2015.

[evilaliv3]

i'm ok with this let's ask for feedback on the various crypto related mailing list and eventually a request to OTF in order to ask for this review; it would be of benefit for all the projects that would like to use a pure python implementation of PGP.

[fpietrosanti]

PyGPGME is heavily undocumented (https://bugs.launchpad.net/pygpgme/+bug/530634 http://pygpgme.readthedocs.org/en/latest/ ) but seems to be:

• a) maintained
• b) part of Ubuntu/Debian package

It could represent a way to fix the mess of python-gnupg exec()uting gnupg system binary.

[evilaliv3]

last edit 3 ears ago (2013-02-13), last release in March 2012 seems to me not that much maintainence.

but it's true that it is in ubuntu and the underlying gpgme is maintained.

https://pypi.python.org/pypi/pygpgme

[fpietrosanti]

@evilaliv3 re-reading randombit mlist it's also part of Xenial with the debian packages actively maintained, made python3 compatibile, there's a bunch of other ubuntu released packages relying on it, so maybe it jus does what it need to do w/out major issues. To be better evaluated/considered

[meskio]

GPGME is not different in the exec thing than python-gnupg, it does exec gnupg for each command. The good thing of GPGME is that is maintained by the gnupg project and might know all the nifty tricks of gnupg better than python-gnupg.

I'm in a similar situation, exploring options for @leapcode.

[evilaliv3]

Thank you @meskio for this clarification.

[fpietrosanti]

@meskio Having that very basic PGP functionalities in pure-python it, being based on OpenPGP-Python of PGPy with basic python-gnupg drop-in replacement/compatible API would be amazing!

[sicurezzadeimedia]

PGP Key Validation are now done using Client Side OpenPGP.js library.

[sicurezzadeimedia]

The only pure python PGP library still actively developed is PGPy:

https://github.com/SecurityInnovation/PGPy/commits/master

[evilaliv3]

@meskio: what is your vision today? what are you using in @leapcode?

[meskio]

@evilaliv3: I did most of the work to port @leapcode some months. There was some small features I was missing, but all the main functionality is in PGPy and it was working fine in my tests. I did manage to remove several hundreds of lines of code in the process.

But I got dragged into other priorities, and sadly we haven't being working on the email part much for a long time. It is still in my backlog to finish that, but we don't have real users yet.

I think today for non-standard email clients (whatever doesn't follow the gnupg/neopg preconceptions) it makes sense to use PGPy. I have some hopes that in the future we'll be able to use sequoia-PGP: https://sequoia-pgp.org/ Which being in rust will make easy to be used in other languages. But for now is a work in progress.

[NCommander]

I've found that PGPy's current release version doesn't work properly on Py2.7 due to a change in six. There's a fix in pgpy's git master but there hasn't been a release in over a year to push it publicly.