SecurityInnovation / PGPy

Pretty Good Privacy for Python
BSD 3-Clause "New" or "Revised" License
314 stars 98 forks source link

NotImplementedError <class 'pgpy.packet.types.Opaque'> when creating PGPMessages from longer blobs #161

Closed seddonym closed 8 years ago

seddonym commented 8 years ago

I've run into a similar issue as reported here: https://github.com/SecurityInnovation/PGPy/issues/160. I think it's might have different causes so I'm reporting it as a separate bug.

The bug is that I get the same NotImplementedError when I run PGPMessage.from_blob() on messages that are not very short. For example:

encrypted_contents = """-----BEGIN PGP MESSAGE-----
Version: GnuPG v2.0.22 (GNU/Linux)

hQEMAz5Gol+BYS0FAQf+IHGdi6zNSXXReOmVS7BH6yrsEoXGH44qbNGIK77eDGfu
bKXssxRcVrsBsYO+en/kY1WV4634zvq5WxtN8WLau+geqBPKwvskFQpFo7kX4eOh
1JBadwcNLEQ01QFEXL087C3OzPha3XYjsZoBMpsa199wDBMPSwNWM7pig7oblkLu
pLaNF1o8CfG1zew0Q8H8fAxLCwhUGE1fI5M0c1qhtbnXxgw08UGuls3ui7cRnh7O
5MuXC3NwBA5/pe8ebBVQ0PgBYhAoEgUkkJ+Vd3VA4JAjWrT+A9S7Iz98WDcY8bzM
EdbR3WTVJEbaa0nRgoMBb2qGWQ/dz3gruZRRjnGN8NLsAWfRnzmXw3cZscEfnAKD
QJUfyr4n7QZjS0FkPoTiS/vQiOttd3D5Sfbh9keiFSEeo2752TqBJIeAAkmai4O+
RBDSIJ2x5g7sHXKKN3nYG0+PSxTkyQgFKerVH90Qw5QqKhSdfB0WvqVjg2PhWEYZ
RArF14sIlkEFuVYEQLiP0saA5wLvMoPgdQ4VzJ7DshqzIlaz6jBiL91od7ApEVeM
mPD/d2N5xF4ypRYn8nTvD33W7Rk9W2JhA/LBrLnQXpYlZMvDxe6oYak6Cg9mAuJ9
F1C5WhoHs81c8Sxt9lc6IsyLHyQnh1jDWVhiKlkUB0Tmu8i8o6IqBhk09I0wZufC
lQ5TORi3YBJa0VhBWqpRPfwiTSc38XgkElEOnfiphUxXio9Tu74zH58/Al/kcouP
NnSzsHDygg6E2dmIng96XgPV0i/mHBPM3z0UUPO/1pATkmAJfR8cmIO1FNQypc9W
oxOU7JVWsHLMlC+ZfzNJ4V3dDa+kMqfqgNoierLQInV8xgcjOSvqehlaBH1iL/gv
YZZCgy1ItglG6CEFFUDE4CT5OtPXaMDOB199ih/3ccZFMyYcWrjruJr3vCD4yqhc
QzG/QcJ3kmK5Wu2xo76lOIKH4dk6i+6ASs6QXh7t/yQOSpGN3V4uq5nSkEx1Glc2
BMGP5l3di3Y2pumvYlwLLFG/4b8pQPaC6jrlbaGzWzUr000a8XuhIMpTt1+w+8Mi
zDy36fhC3YGPlAWJqg9QceB5UV+57k+fTyU9qAzRfNihn5p3qZFILSZldnFYUZuc
qmfwioAsr3zvI9Rjg32pKtSOiHgz4jxHT3uf1mCPnjjQ1zOPJjqi0Y+uqAg/5mL4
bjt0SmigkFBS6rUNKYMq8y1mpXuFlNdrrCZS5nuszMwc8OM4Q/9+j2Ls7Qxx7IHn
E+kv6Pfg3LDcNiH+fU8n2CqRiTfIQCjJpEiunNDQL7izvukNxCzkKeiBl5ASuQC0
qmZGb6LagXi/PWTL0Dvy8P79c/vPEWuBgXdErzHEdMRijgepdW+MXOxgvIGfq7RT
CiApDJPpMynILfQYz8d7IRQXDRM5WzDPHpuGtT30IfTsboDIxE+iEc5kmhv2VDhr
OeJqTkJuNjxAz0zhf1HRCt15FIoxTNKA8ITABfvrpKeNXwDvY/bNCWnTmjSi4pI/
LRub9HN27eRiBrtQnDb96xDmD1Q0ZDN2aY7O8rLp9wmiL5O+FN6RbKQEFURrcKsm
LYgQ0u3rDVOoOefPyunk31jU2SEYoSEFyFV9/Nyh1sahPz9xdT+RVx5UperR7Ab9
kwfhYKHsVvy2Diq1h5Ya+pIlkL5YIHo1BmB65Dna0WWCnDGWtOLo0HQx9+E8nX/p
LQOGIsEtnXPBpKQxN5PiHgUePY2aBujfzmU3JN1eLhgIbbOKMt3G+SWZTbSN4/Xt
YBp43SmI5VnV6LRwh5LAA4umDpEIJYoFegOGsMMnSmJ89E+69X+1g5BJ9OhMVjq7
Zo3Nu1oPb2/AEXcq1JCYFqOmdj27f2f6PtXW+N0RXq+UlTHBAp3nL7xmyvZJSa4B
RkcTzI52LdzEWcEbT7FVEOW01QJd9VqOBAx2SHRrJLT1FCb7DCSSJAkMje8o6WWO
ge3Yeqb/QJ2sGOcIAM+dJjn1UMFRxCgRBUAswjjzJRkXnGKbMYVbyT9vuP1fdKIs
BIj66k3CNGGS6Yz9lSFs+cGRO693hiP6pSEY0VsY0fq394UrMNmvUJcvkfhNCFr6
XMNKFqM3uDt1Vnuec5q7LG4W2921pCx7fmCVDR3hH1FsI3olxLE6WP/RP5FXEIT8
KZ1My8dFbbAlzbx+KL7IWxHkI6Zu8HD9w81U2MP+rnfVk3hd62fAlfZrDuZtcrvs
7tQDV3RrD8l3ANcGRd6POV3PesOVV5TeeotPZE1+aY32smBYwJCZ1a2LSktfjf7C
mjRVLKgfyGfIdul9AXeHcdTV/soVs+gDmGx0IEHyTwjeP7NokR0opS0XgcuEwwid
lo8DH4Zfe+seSSHZT12t9BGCTEtoies7WICS96FF1S3RmAmXtSlhIdW1YJEONUzZ
SqHIZPXwoTQo8i8BSqustMKRYsDapFZ99fcUr5KWlC+vc5mUpjwJv0egNVYSOwu6
5QHJVQ2RbsG+RjA2VOLUlkPhim7NeJ39DFFQqxiYrUluEOtVDbcF38hjhG4CHMel
9KWfdqJHnuaL+bJIE3GHEkFMhvbBkyfS2VXbiUuOWh4QSmHXfbzrIFTFr2j5Wp6x
c4APy8j6Xlk0IEn9izRHCBhL7ZBWZmnND1koBoieMIceUwLf6FNtpTuSSKt5Hv0Y
7vJkyLVvqfmqbpuXv1lJUtlevfarly5Ob5eZDGzd1iPqtEJkmPrjQ63kDyyiLcXN
jNdoNDCELbhTZWHman5IxgKeR5IGeJQ5yh1Pd8dv07f3XclcWdBhMe0TkLG4XJot
ePzZgYPnb6A2X4sS9PEXnuRFwQudaeHqg3bnMVlena0Kx0+EMPilOh9fKDsVY2X1
NMsE346PlF3ZsC5Qp8t+fqCQ+lTJSgY3rnKbBH0gwuMFSVxFqwqDEbpJIm+MVZhA
8Zt/k/y/GPFNTXqDQlwF2jyupHHYXpaoiWC8kcenkiAksZdIYr0xb7SFHXHPHB2C
QnCmW2dGxheGrYeGuh6jXNjrHUF+oGxehvSmP2ttVmr6DFim0WYeVkXXpTgFBbGU
1pDCYbPuBoCL3LdMTIFQIf2PPa6KDnZ4cGkFzbAHWwRmY6I3ljiooJkNbwboGkgG
IEoZCTJlLuMhc2HbkeNsQdvo4DbOmOIumJiQqwT2Z4NmLR4AzcN4LTP7M/MdYK1B
9xBrpFoZR+kfqBot/jOSG2+Uuhl4RY5DlbfPoGvStjWmzqb0e0yag5UTpmoibvh9
Z3+LcoDgr3puTzo/DZG86pb8ZDZ5qFxniCga0KQnWpuu34f1k5/fEbEPgJCXBkYK
CE8hVR1oj1QQ2JFAUwp1cimMib5SrXIfNYlVHQ29I84CAr+kOdh1gFx2poIND8CL
qqx5Koga9O7WEPs+bC/S4ONGlZNw8UPIrQp6X1HxeCAhzVyd8Bnwt683oWd72RzM
y/h5XWj7ca2+77PE/4GFUsD4M8P8w0keUxAUDWQmwH0HfoORDIVaiCS/8E47O9zr
HtWNNPQ8gJ4Vuj1rksvhZNY3KJW7dtnraGuLa4j+Qc4XWhGOtTTsiHiwFRZazWC+
nKUl01hDfFVT54fXjl4nJqjRTL/eXQBst6ZSEkb3ZYfzdQ6vVL0DvXIopbN1AihB
EUnxcb+lxyVXCdoTWIYHnUZakXv+2yUO5o3MosrYzV7fl4KaAEj8+R7Mz0EvtqkR
+Yq/mrflO2C14f+vHwuRYmITEmsa62BhYZ9DneQroP8wPVtq7EtSZwHk4GV+ziLR
pWWje3wxkd/fjK0u31XbDtw9XVLkeadO3luT6CgJujOgY2ShFn3tVi7QCxYOrMCf
G6NrHI/4D8dIWXQrmrBVIAMQzeUHoa9go5eZE1ym2wZ4Q7QWyzpldPJOoK8ZrInG
8YRzxh9uABMHIDtUJ2QPcGCJfHlygn7pKifaIUWRvlE4bCDTQ9aHS75g1QAKvCDE
CHkHw/w+3GUan7DwtFrSxkG8a1d6D3z/f27x4x5xn6ijhlcQsdzAHsK3E9roPtwH
wgk/OGkUQCfuBIcWGUwIOmB4KqGnqkMaU+egi9rq6VypP8z17dS1SiWBAwJL2PGR
6KjAGAfrqqIx2+LBpHAwyylyKfba2FvLVbahVg/v/6DRe5IKFA7FfAI3uhK2lD54
uYv9TjAbIwrt79zkdYmQExGbCHWMNDHKzLh2iMpZZaWyBINirUx1kiCVd1dFdxER
am+ex9I1vcFB2hQKvQZjnBihD30ESpzpRC508OsOJfvAltdQhWEnIcgiKMUBcR8T
mYGL2k/zqioWIhyudFsumOV2EesAXuO9oMSChprIhunnEHE0vTRaxCEs0Gg5+MF9
fb7PvHjSKYHHfOKXAKNofHLwdJDJWAbIj1IUUO0eFSHrUhPBM/F2DzaBQJNqLVKt
tAqFpBRiYvFB6gGkhZ5tPOUbnRLMi6qQ4jZTvWlynDiIEJSxPPHcotb+X0PHTvYZ
FiMQIfRmXkPIWFW4nvomm1zJAKz1eVqycqsDlJeU/ifnKfmIqpm4QeSSD6GSOKnr
eKe5rbtu5RFB6KWbOLW6jetORQK4ZceUtqPnps4l1CYWYzC9Otg4KkZONM1aDKfN
0MSB1yh8eG8hfhWmsmktpFlOK5BD45O9swidyZvADnZy66aHPUPInjvduzuMotNv
2sjp2E6beTGs3mbVEoMRxvu0P8K9087amxpGxdKK1kNua3RdomcX8blC7gOh8szh
emcbVbG4MezQcj9pEtz2qPYXDJi30j7axXwryB/lC8BvOUlEWqGaYVrldhzcVwmc
lmiSPDDXm80B/3zK1iYZwz3kRxQRD248/UN61sCJ1Zt8RM0tTB1pQ61oHO2nXYuJ
q/isJd7FkfBD8BjQYjiKP3N1pRRwGdvfZ1r2YhK/3sX0pKQPSh0cI+dlWnqzhu5y
8O1FongIMfxXBzv2R85rll/UizrTMhbxW0G4VjbflIc5p2wwqj0uAQQG3X4qtO0F
VcS3xEQudBDvtQ2k31dMN2zzr44DuxdUwhSXISKkk3OKwEU11opvOhY2/anzkkmH
qxdKJOrduq3M8ce7NML75J5Roc881wCy7Zdw6CpAwQNl/CV5dzwxc7WF56PfSXzD
7NZjBMvalGLsizj9qA5rDS5IzYU49VYXXF3F4eSL+yhaaHyw/CgTwYhtOCSCXSyZ
l57KK33rFuX4bR3gfvm+VT/yVG7jjitmIzoHnLqH8liFEkOnHwxV6XwwgG/Bte1V
7caJWjO56o773zGIV0JlkHEFbUnAa+ENhf528Cu2aF6xs+WTF8oLdNdRHNuW3swQ
r0HM1cSD+kUqcp2SVQg3sHr8nnE2E2OG10/8Ztk/FVffBsHGgoqQbZkw9Dto9NjD
CHEpO0nYeY/kVECvROB2kHnpYJKy753tzS/hCYGHjybh8/GQXa9y4Y4Srojnh1Hb
4hNcLs0Obaur4DnwQyoCB/nytU1AQesNUnfxwsGpmH6g7XVt1uyS2NCKEPbLyexK
vE+v4fjIaP1VDM04LsQH0UEj8X7Go8aQE4jAK1RXUUqeuOEOcf6JiARfpU0jKrFk
pS/E5urMss/jPpyxkmC2wkThYrsSdOSA5PXVk0/mXiXky4DMuFjc1ysrGBc60+Fk
TbPwB7zi3OA5G+JTgy+qqd/zl7InwwPKcILt2kBwsvif3J9qDBYuVwnmcXNVjVfd
yvTjnYEGlv28FnxgNyINrslIc2X9Pm+bMYy+oazvvrm9g/jLLq03zM13Tjs8u8nB
RYs0cYX4lpqbm9wyL4/AYavmzfFWecnpHLTFFhHKUrr5Yhqkk3n8uCvO3RRq9rRM
DVZF5lxJe/CFFqTdP5fUcTFex8NE+8YlR7uLiZWsFMYtlVaMfoL1EFHBkB9UkAfE
Xunsw/rYIXHT4GaBBg99AqsJSn7a6TNPOPNsvd3BJPZvY2lN6vTRew==
=aF0T
-----END PGP MESSAGE-----
"""
import pgpy
pgp_message  = pgpy.PGPMessage.from_blob(encrypted_contents)

This raises the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/xxx/local/lib/python3.4/site-packages/pgpy/types.py", line 187, in from_blob
    po = obj.parse(bytearray(blob, 'latin-1'))
  File "/xxx/local/lib/python3.4/site-packages/pgpy/pgp.py", line 1084, in parse
    self |= Packet(data)
  File "/xxx/local/lib/python3.4/site-packages/pgpy/pgp.py", line 874, in __or__
    raise NotImplementedError(str(type(other)))
NotImplementedError: <class 'pgpy.packet.types.Opaque'>

This is a message that consists of 1,000 lines of random 7 digit integers, one on each line. I have no problems with loading a message that is just 100 lines.

$ gpg --list-packets message.txt 
:pubkey enc packet: version 3, algo 1, keyid 3E46A25F81612D05
    data: [2046 bits]
:encrypted data packet:
    length: unknown
    mdc_method: 2
gpg: encrypted with 2048-bit RSA key, ID 81612D05, created 2016-06-15
      "Xxxx Xxxx <x@example.com>"
:compressed packet: algo=2
:literal data packet:
    mode b (62), created 1466091758, name="",
    raw data: unknown length
Commod0re commented 8 years ago

Ah, this is a case of PGPy not having support for partial packet lengths just yet (see #95), but it is something I am planning to implement for the next release.

Depending on how you invoke gpg, it will chunk the data packets using partial packet length headers so it can stream the encrypt/decrypt operations while still receiving input. That looks to be what has happened here:

% pgpdump 161.asc
Old: Public-Key Encrypted Session Key Packet(tag 1)(268 bytes)
    New version(3)
    Key ID - 0x3E46A25F81612D05
    Pub alg - RSA Encrypt or Sign(pub 1)
    RSA m^e mod n(2046 bits) - ...
        -> m = sym alg(1 byte) + checksum(2 bytes) + PKCS-1 block type 02
New: Symmetrically Encrypted and MDC Packet(tag 18)(4096 bytes) partial start
    Ver 1
    Encrypted data [sym alg is specified in pub-key encrypted session key]
        (plain text + MDC SHA1(20 bytes))
New:    (86 bytes) partial end

I generated a similar file (but slightly larger) and tested a bit like so:

 % stat -c "%s" data.txt
8200
 % wc -l data.txt
1025 data.txt
 % # the options I used here were used to skip compression and encrypt with
 % # just a passphrase for easier demonstration purposes
 % gpg --armor --symmetric --cipher-algo AES --compress-algo 0 data.txt
 % pgpdump data.txt.asc
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
    New version(4)
    Sym alg - AES with 128-bit key(sym 7)
    Iterated and salted string-to-key(s2k 3):
        Hash alg - SHA1(hash 2)
        Salt - bf 72 81 ce 4f a1 38 eb
        Count - 31457280(coded count 238)
New: Symmetrically Encrypted and MDC Packet(tag 18)(8258 bytes)
    Ver 1
    Encrypted data [sym alg is specified in sym-key encrypted session key]
        (plain text + MDC SHA1(20 bytes))
 % # now I'll pipe the contents of data.txt into gpg instead, 
 % # which will cause it to stream
 % cat data.txt | gpg --armor --symmetric --cipher-algo AES --compress-algo 0 > piped-data.txt.asc
 % pgpdump piped-data.txt.asc
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
    New version(4)
    Sym alg - AES with 128-bit key(sym 7)
    Iterated and salted string-to-key(s2k 3):
        Hash alg - SHA1(hash 2)
        Salt - b9 18 9b e3 79 21 34 27
        Count - 31457280(coded count 238)
New: Symmetrically Encrypted and MDC Packet(tag 18)(8192 bytes) partial start
    Ver 1
    Encrypted data [sym alg is specified in sym-key encrypted session key]
        (plain text + MDC SHA1(20 bytes))
New:    (58 bytes) partial end

I was able to force it to not use partial packets by adding the --set-filesize option like so:

% cat data.txt | gpg --armor --symmetric --cipher-algo AES --compress-algo 0 --set-filesize 8200 > piped-sf-data.txt.asc
gpg: Note: --set-filesize is not for normal use!
 % pgpdump piped-sf-data.txt.asc
Old: Symmetric-Key Encrypted Session Key Packet(tag 3)(13 bytes)
    New version(4)
    Sym alg - AES with 128-bit key(sym 7)
    Iterated and salted string-to-key(s2k 3):
        Hash alg - SHA1(hash 2)
        Salt - c3 c8 2e 8b 60 54 1e fe
        Count - 31457280(coded count 238)
New: Symmetrically Encrypted and MDC Packet(tag 18)(8250 bytes)
    Ver 1
    Encrypted data [sym alg is specified in sym-key encrypted session key]
        (plain text + MDC SHA1(20 bytes))

Since partial packet support is on the roadmap, I'll close this issue for now.

seddonym commented 8 years ago

Hi Michael,

Thanks so much for your swift and detailed reply, this is very helpful.

I'll keep an eye on the partial packet support, and will try the workaround for the time being.

David

seddonym commented 8 years ago

Incidentally I wonder how difficult it would be to improve the error messages for features that are known to be unsupported. Perhaps a simple way would be to have a page in the docs which lists known encryption formats that don't work, and the exception could point them there. Even better would be for the exception to say what feature it doesn't support - though I imagine that would be more work. Just a suggestion, might save people time.