Able to decrypt file in regular pgp but not using PGpy

[pylix]

I'm able to decrypt a particular file using the following command in gpg

gpg --batch --yes --skip-verify --passphrase=$passphrase -o $dest -d $src

however when trying to decrypt this file using PGPy I get the flowing error.

Error <class 'pgpy.packet.types.Opaque'> Traceback (most recent call last): File "/var/task/lambda_function.py", line 405, in lambda_handler out_path=out_path) File "/var/task/lambda_function.py", line 291, in decrypt dec_file_data = priv_key.decrypt(encrypted_file) File "/opt/python/pgpy/decorators.py", line 129, in _action return action(_key, *args, **kwargs) File "/opt/python/pgpy/pgp.py", line 2490, in decrypt return self.subkeys[skid].decrypt(message) File "/opt/python/pgpy/decorators.py", line 129, in _action File "/opt/python/pgpy/pgp.py", line 2499, in decrypt decmsg.parse(message.message.decrypt(key, alg)) File "/opt/python/pgpy/pgp.py", line 1281, in parse self |= Packet(data) File "/opt/python/pgpy/pgp.py", line 1027, in or self |= pkt File "/opt/python/pgpy/pgp.py", line 1068, in or raise NotImplementedError(str(type(other))) NotImplementedError: <class 'pgpy.packet.types.Opaque'>

Even though the file decrypts successfully in using gpg command line I'm getting this notice when inspecting the file

gpg: encrypted with 2048-bit RSA key, ID 38E5225D1330EE11, created 2006-02-16 "x@mycompany.com x@mycompany.com" gpg: Signature made Wed 09 Feb 2022 05:54:34 AM EST gpg: using DSA key 25E1E598435F4639 gpg: Good signature from "Y Commercial y@companytwo.com" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 864F 4D2E B677 2B9D 3018 68AA 25E1 E598 435F 4639 gpg: WARNING: message was not integrity protected gpg: decryption forced to fail!

Other files encrypted with this public key from our other clients successfully decrypt using PGPy, but this one is not. Is this expected behavior? is there a workaround for forcing the decryption of this kind of file in PGPy?

[J-M0]

Hi @pylix, can you post the code you're using that causes this error please?

[Commod0re]

whenever NotImplementedError: <class 'pgpy.packet.types.Opaque'> shows up that typically means one of two things:

1. the file contains a packet type that is not supported by PGPy
2. the file is malformed, or otherwise does not parse correctly in PGPy (possibly due to a bug)

admittedly, the exception message could be improved by including the parsed packet tag, which would also provide a better hint for which one it probably is

---

the timestamp on the message provided in the gpg inspect output being in the year 2006 suggests it's probably an older style, RFC 2440 message, where PGPy is mainly focused on RFC 4880 compliance (RFC 4880 dropped in 2007)

What we'll need to debug this further is either:

• a dump list of all packets in the message, which can be generated with one of the following:

  • pgpdump <file>
  • gpgp --list-packets <file>

  a spec for the key so we can generate a similar one for testing would also be useful

• a PGP Message and key we can use to reproduce this issue with if you go this route, please generate a key and message specifically for us to use for debugging and regression testing

[pylix]

Below is the pertinent code. note that this method works on other files encrypted with the public key coming from other sources but is failing one this specific file from this specific data source

import os
import base64
import re
import json
import gzip
import io
import logging
import sys
import gc
import traceback

import boto3
from botocore.exceptions import ClientError
from boto3.s3.transfer import TransferConfig

import pgpy

def decrypt(s3_file_path, keep_encrypted_file=False, out_path=""):

    # reading from s3 into mem
    etl_s3 = MCP_ETL_S3()
    s3 = etl_s3.s3
    s3_client = etl_s3.s3_client

    bucket = s3.Bucket(etl_s3.s3bucket)
    print("Bucket is", etl_s3.s3bucket)
    print("Object to decrypt is", s3_file_path)
    object = bucket.Object(s3_file_path)

    file_stream = io.BytesIO()
    object.download_fileobj(file_stream)

    file_in = file_stream.getvalue()

    priv_key_mem = (
        get_secret("/key1/$ENV".replace("$ENV", ENVIRONMENT))
    )

    passcode = (
        get_secret("/phrase1/$ENV".replace("$ENV", ENVIRONMENT))
    )

    # Because the passphrase is bytes, we need to convert it to string 
    # Because this is what unlock is expecting
    passcode = passcode.decode('utf-8').strip("\n")

    priv_key, priv_key_others = pgpy.PGPKey.from_blob(priv_key_mem)

    encrypted_file = pgpy.PGPMessage.from_blob(file_in)

    with priv_key.unlock(passcode):
        dec_file_data = priv_key.decrypt(encrypted_file)

[pylix]

@Commod0re I masked the names of the entities by masking the alpha characters with x's, but kept the lengths and casing the same.

$ gpg --list-packets file_J_2022_02_09.pgp
gpg: encrypted with 2048-bit RSA key, ID 38E5225D1330EE11, created 2006-02-16
      "xxxxxx@xxxxxxxxxxxx.xxxxxxxxxxxxxxxxx.com <xxxxxx@xxxxxxxxxxxx.xxxxxxxxxxxxxxxxx.com>"
gpg: WARNING: message was not integrity protected
gpg: decryption forced to fail!
# off=0 ctb=a8 tag=10 hlen=2 plen=3
:marker packet: PGP
# off=5 ctb=c1 tag=1 hlen=3 plen=268 new-ctb
:pubkey enc packet: version 3, algo 1, keyid 38E5225D1330EE11
        data: [2047 bits]
# off=276 ctb=c9 tag=9 hlen=2 plen=0 partial new-ctb
:encrypted data packet:
        length: unknown
# off=296 ctb=c8 tag=8 hlen=2 plen=0 partial new-ctb
:compressed packet: algo=2
# off=299 ctb=c4 tag=4 hlen=2 plen=13 new-ctb
:onepass_sig packet: keyid 25E1E598435F4639
        version 3, sigclass 0x01, digest 2, pubkey 17, last=1
# off=314 ctb=cb tag=11 hlen=2 plen=0 partial new-ctb
:literal data packet:
        mode t (74), created 0, name="XXX_XXX_XXXX",
        raw data: unknown length

If you think from the output above the file is malformed we can close this issue. Somehow it's able to be decrypted with gpg in spite of this. Unfortunately I'm not privy to any of the key information besides what's listed here (the fact that it's an 2048-bit RSA key) also it's protected with a passphrase

[Commod0re]

aha, I see

so the outer message is well formed, but looking at both that and your stack trace, I can see that it's failing here, on line 1027 https://github.com/SecurityInnovation/PGPy/blob/02766befcd5ee6c3a5d3ec0c8b46aaea1f9bb30c/pgpy/pgp.py#L1024-L1028

which indicates it's having trouble parsing something inside the decrypted compressed data packet

if possible, could you decrypt the message with gpg, and then run gpg --list-packets on the decrypted message?

[pylix]

@Commod0re After decrypting the file I was not able to run the gpg command on the resulting data

$ gpg --list-packets file_J_2022_02_09
gpg: no valid OpenPGP data found.
gpg: processing message failed: Unknown system error

I was able to get info on the key The key is AES: CIPHER_ALGO_AES = 7 gpg: AES encrypted data

And also additional info on the signature

gpg: original file name='XXX_XXX_XXXX'
# off=29969976 ctb=c2 tag=2 hlen=2 plen=63 new-ctb
:signature packet: algo 17, keyid 25E1E598435F4639
        version 3, created 1644404094, md5len 5, sigclass 0x01
        digest algo 2, begin of digest c3 aa
        data: [159 bits]
        data: [158 bits]
gpg: Signature made Wed 09 Feb 2022 05:54:54 AM EST
gpg:                using DSA key 25E1E598435F4639
gpg: using pgp trust model
gpg: Good signature from "Y Commercial <y@companytwo.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 864F 4D2E B677 2B9D 3018  68AA 25E1 E598 435F 4639
gpg: WARNING: message was not integrity protected
gpg: decryption forced to fail!

[Commod0re]

Hmm... 🤔 oh, right, I think gpg only saves the message contents when decrypting, not the full set of decrypted packets, that will make this a bit more difficult to debug

Unfortunately I can't really guess at what the actual problem is without knowing a little more about what's in there, and PGPy won't tell us what it's choking on as-is. I'll have to think about this a bit, I'll probably have a script for you to run on that message file tomorrow to try to get the information I need to see what's wrong and create a regression test and (hopefully) a fix

[pylix]

@Commod0re I'm closing this issue. We believe root cause was that we did not have the signing key in the keying on the system in question. The reason we know this is because decrypting the file and then re-encrypting it with the same key without the signature solved the problem.

As a side note, we also saw issues where PGPy went into an infinite loop when trying to decrypt a larger file. The root cause here appears to be related to fact that there is sometimes some unpredictable behavior when using these older RFC 2440 messages. We solved the never ending execution problem by generating a new key pair for this use case ensuring we're using the latest RFC 4880 messages. We also switched to using ASCII armored files instead of binary but I don't think this is related to the success when using the newer keypair.