[2.0.2] decrypt failure, works on 1.4.0

I've a fairly stable system. For the most part, save security patches and a few minor quality-of-life upgrades, it package state is untouched. I've been using gnupg-1.4.0 for over a year and decided to upgrade to gnupg-2.0.2. In testing, I discovered that my application could no longer decrypt messages. I've included the failure and success cases below.

$ /usr/bin/gpg --version
gpg (GnuPG) 1.4.16
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA-E, RSA-S, ELG-E, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

What was working under gnupg-1.4.0 no longer works on gnupg-2.0.2. Here's the gnupg-2.0.2 reproduction case.

$ sudo python3
Python 3.5.2 (default, Jun 27 2016, 18:58:10)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gnupg
>>> gnupg.__version__
'2.0.2'
>>> gpg = gnupg.GPG (homedir='/var/firefly/gpg', binary='/usr/bin/gpg')
>>> re = gpg.encrypt (b"You can't take the sky from me", '503C0CFE6CC34E1D296F0771C5A1D2380FF007B3', armor=False, compress_algo='Uncompressed')
>>> re.ok
True
>>> rd = gpg.decrypt (re.data, passphrase='1a1b8f347301981f5e99a8bd6b50446efff12e361363518f00bca98c6b3ff982')
>>> rd.ok
False
>>> rd.status
>>> rd.stderr
"gpg: WARNING: unsafe permissions on homedir `/var/firefly/gpg'\ngpg: no valid OpenPGP data found.\n[GNUPG:] NODATA 1\n[GNUPG:] NODATA 2\ngpg: decrypt_message failed: eof\n"

And here's the gnupg-1.4.0 success case using the same statements above, created in a virtual environment.

$ virtualenv --python=python3.5 gnupg-test
$ pip3 install https://github.com/isislovecruft/python-gnupg/archive/1.4.0.tar.gz
$ pip3 list | grep gnupg
gnupg (unknown)
$ sudo bash
# . gnupg-test/bin/activate
# python3
Python 3.5.2 (default, Jun 27 2016, 18:58:10)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gnupg
>>> gnupg.__version__
'unknown'
>>> gpg = gnupg.GPG (homedir='/var/firefly/gpg', binary='/usr/bin/gpg')
>>> re = gpg.encrypt (b"You can't take the sky from me", '503C0CFE6CC34E1D296F0771C5A1D2380FF007B3', armor=False, compress_algo='Uncompressed')
>>> re.ok
True
>>> rd = gpg.decrypt (re.data, passphrase='1a1b8f347301981f5e99a8bd6b50446efff12e361363518f00bca98c6b3ff982')
>>> rd.ok
True
>>> rd.data
b"You can't take the sky from me"

In _copy_data, the routine binary is messing up the input data.

>>> coder
<codecs.CodecInfo object for encoding utf-8 at 0x7fffefaea888>
>>> len(data)
593
>>> _py3k and isinstance(data, bytes)
True
>>> len (encoded)
1076

It's taking a bytes string and decode/encoding it and creates... something; the operation is not idempotent. If in _copy_data, I change the code to:

        sent += len(data)
        #encoded = binary(data)
        encoded = data

it works like a champ, both encrypting and decryptingbytes data type.

When examining the operation on the bytes plaintext string, the binary routine is idempotent (in _copy_data):

>>> data
b'Encrypt this!'
>>> encoded
b'Encrypt this!'

However, mix a little un-decodeable characters in there and binary goes bonkers.

# The test data to encrypt...
>>> b = bytes([i for i in range(0, 255)])
>>> b.decode ('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 128: invalid start byte

Sending b into gnupg (in _copy_data):

>>> len(data)
255
>>> len(encoded)
509

I don't grok Python's codecs, and won't claim to know the fix. I'm surprised this garbled the byte-string instead of raising the UnicodeDecodeError (in binary):

>>> data
b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe'
>>> coder.name
'utf-8'
# Didn't throw the UnicodeDecodeError...
>>> data.decode(coder.name)
'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f�������������������������������������������������������������������������������������������������������������������������������'
>>> coder.encode(data.decode(coder.name))
(b'\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd', 255)

I even tried data.decode(coder.name, errors='strict') (reinforcing that keyword default value) with no change in behavior.

isislovecruft / python-gnupg

[2.0.2] decrypt failure, works on 1.4.0 #156