PY3: 2.01 - gnupg stream decode error when decrypting without asciiarmor; gpg --decrypt works

[doktorstick]

In a direct upgrade from 1.4.0 to 2.0.1, I'm seeing stream decode errors. I'll preface this by saying that I didn't check to see if the stream decrypt interface changed. I validated that I could decrypt it from gpg command-line.

niska@firefly3$ sudo python3
>>> import sys
>>> sys.version
'3.4.0 (default, Apr 11 2014, 13:05:11) \n[GCC 4.8.2]'
>>> import gnupg
>>> gnupg.__version__
'2.0.1'
>>> gpg = gnupg.GPG(binary='/usr/bin/gpg')
>>> kwargs = dict (armor=False, compress_algo='Uncompressed')
>>> renc = gpg.encrypt (b'Dance like a psycho', '47089192', **kwargs)
>>> renc.ok, renc.status
(True, 'encryption ok')
>>> rdec = gpg.decrypt (renc.data, passphrase='streamtest')
>>> rdec.ok, rdec.status
(False, None)
>>> rdec.stderr
"gpg: no valid OpenPGP data found.\n[GNUPG:] NODATA 1\n[GNUPG:] NODATA 2\ngpg: decrypt_message failed: eof\n"
>>> with open ('test.enc', 'wb') as fo: fo.write (renc.data)
...
599
>>> ^Z
[1]+  Stopped                 sudo python3
niska@firefly3:~$ gpg --version
gpg (GnuPG) 1.4.16
[snipped]
niska@firefly3:~$ file test.enc
test.enc: GPG encrypted data
niska@firefly3:~$ sudo gpg --decrypt test.enc
passhprase: streamtest

Dance like a psycho

[isislovecruft]

@doktorstick I added your example code with some minor modifications as a new test in the test suite… but I can't seem to reproduce the issue, on either Python2 or Python3. Do you think you'd be able to build from my fix/102-py3k-decrypt-bytes-literal branch and run the tests? E.g. make install && make test (works in a virtualenvs too).

[doktorstick]

Sure, no problem. My results are below. TL;DR version, armor=False is not working in my test case.

I created python3 virtualenv and cloned the branch.

niska@jayne02 virts$ virtualenv --python=python3 fix-102
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in fix-102/bin/python3
Also creating executable in fix-102/bin/python
Installing setuptools, pip...done.
niska@jayne02 virts$ workon fix-102
(fix-102)niska@jayne02 virts$ cd ~/code/branches
(fix-102)niska@jayne02 branches$ git clone -b fix/102-py3k-decrypt-bytes-literal https://github.com/isislovecruft/python-gnupg.git
Cloning into 'python-gnupg'...
remote: Counting objects: 4016, done.
remote: Compressing objects: 100% (35/35), done.
remote: Total 4016 (delta 18), reused 0 (delta 0), pack-reused 3980
Receiving objects: 100% (4016/4016), 2.78 MiB | 2.64 MiB/s, done.
Resolving deltas: 100% (2484/2484), done.
Checking connectivity... done.
(fix-102)niska@jayne02 branches$ cd python-gnupg/
(fix-102)niska@jayne02 python-gnupg$ git status
On branch fix/102-py3k-decrypt-bytes-literal
Your branch is up-to-date with 'origin/fix/102-py3k-decrypt-bytes-literal'.

nothing to commit, working directory clean

I did the make install && make test in the virtualenv context and was crit by the wall of text. But, I'm guessing the bit you are interested in is:

12899 L1009:decrypt_file       DEBUG   decrypt result: b'Dance like a psycho'
ok

Looking at the output I noticed that you are armoring the output for display purposes.

I ran a slight variant of the code I posted when I opened the bug; the passphrase is a bit longer as it was a test one I made the other day. Note that this block of code causes the problem.

(fix-102)niska@jayne02 python-gnupg$ python3
Python 3.4.0 (default, Apr 11 2014, 13:05:11)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gnupg
>>> gnupg.__version__
'unknown'
>>> gpg = gnupg.GPG(binary='/usr/bin/gpg')
>>> kwargs = dict (armor=False, compress_algo='Uncompressed')
>>> renc = gpg.encrypt (b'Dance like a psycho', 'A0FBD1F0', **kwargs)
>>> renc.ok, renc.status
(True, 'encryption ok')
>>> pp='c80721688450cec9f2693d8b20af657cc84bc27aeab1913f765ce8aaa28457fd'
>>> rdec = gpg.decrypt (renc.data, passphrase=pp)
>>> rdec.ok, rdec.status
(False, None)
>>> rdec.stderr
"gpg: no valid OpenPGP data found.\n[GNUPG:] NODATA 1\n[GNUPG:] NODATA 2\ngpg: decrypt_message failed: eof\n"
>>>

Now, if I run the same code but instead set:

>>> kwargs = {}

it works. For bonus points, I also did kwargs = dict (armor=False) (decryption failed) and kwargs = dict (compress_algo='Uncompressed') (decryption passed) to narrow down the option that was causing grief.

[isislovecruft]

@doktorstick Interesting! Thanks for looking into this! I'll add more tests. I also already changed the debug output (in a749acf4) to only log if armor=True, to avoid breaking the terminal if a test uses armor=False.

[Alir3z4]

I have the same issue as well. Python 3 doesn't work.

[e18r]

This issue has been open for two years. Any updates? Any alternative GPG packages for python?

In the meantime, I overcame this issue by encrypting and decrypting only in armor mode.

[nixwiz]

I have the same issue currently when attempting to use decrypt_file with Python 3.6. It works fine with Python 2.7.

[instantname]

In my case, the tiny change below fixes the issue:

--- a/gnupg/_util.py
+++ b/gnupg/_util.py
@@ -200,9 +200,7 @@ else:
 def binary(data):
     coder = find_encodings()

-    if _py3k and isinstance(data, bytes):
-        encoded = coder.encode(data.decode(coder.name))[0]
-    elif _py3k and isinstance(data, str):
+    if _py3k and isinstance(data, str):
         encoded = coder.encode(data)[0]
     elif not _py3k and type(data) is not str:
         encoded = coder.encode(data)[0]

[meekn]

@instantname I've been suffering from the same issue and your solution works. Thanks a lot!

As far as I can see there seems not to be any pull request made for it. Do you have any plan to do?

[instantname]

@meekn Unfortunately, I don't have a knowledge of the code good enough to be able to say that the fix does not break something else. In particular, there must be a reason why, in the original code, the "if" condition had two cases for what concerns Python 3, and I don't know what that reason is. That is why I did not write a PR.

[meekn]

@instantname Thank you for your reply. I see the reason. @isislovecruft If you see this thread, It would be appreciated if you could give some opinion about the patch. The original code seems to decode-and-encode the byte stream with specific character code (utf-8 default), and it ruins the byte stream.

[haadr]

Thank you @instantname for the patch!

I recommend simply creating the pull request and seeing what feedback you get, otherwise this might no receive any attention, which would be a shame seeing as it seems to plain not work with python 3.

[instantname]

@haadr Done!