Integration tests fails with PGP interop for ECC keys ( gpg: decryption failed: No secret key)

neuhalje commented 5 years ago

Describe the bug The integration tests with gpg fail for exporting BouncyGPG generated private ECC keys to gpg.

To Reproduce Failing test for ECC keyrings, although RSA keyrings work.

Expected behavior The tests should succeed

System:

OS: MacOS
Java [output of java -version]: openjdk version "11" 2018-09-25
Version [e.g. 2.1.2]: commit 3af038f
gpg: gpg (GnuPG/MacGPG2) 2.2.10 with libgcrypt 1.8.3

Additional context

The exported keys:

cat cmd_001_log-ImportCommand/to_gpg_stdin.log | gpg --list-packets -v
# off=0 ctb=94 tag=5 hlen=2 plen=119
:secret key packet:
    version 4, algo 19, created 1546083299, expires 0
    pkey[0]: 082A8648CE3D030107 nistp256 (1.2.840.10045.3.1.7)
    pkey[1]: 0445B0F04963E49A626B0CAC27908A9064442E2D829E2ACF5179260CA12427039BC1FE0460DA1D0BD6EEE6334973AC918BE50A99DF7831B55F6CA200752D2E18DA
    skey[2]: CDB24D40EF9DE69DC60FFAEC70C504BBE3DCF5DB2C1A7B146710C22DBD99B543
    checksum: 11e3
    keyid: 732D8515470359BC
# off=121 ctb=b4 tag=13 hlen=2 plen=35
:user ID packet: "Juliet Capulet <juliet@example.com>"
# off=158 ctb=88 tag=2 hlen=2 plen=123
:signature packet: algo 19, keyid 732D8515470359BC
    version 4, created 1546083299, md5len 0, sigclass 0x13
    digest algo 10, begin of digest 76 82
    hashed subpkt 2 len 4 (sig created 2018-12-29)
    hashed subpkt 27 len 1 (key flags: 23)
    hashed subpkt 22 len 4 (pref-zip-algos: 3 1 0 2)
    hashed subpkt 11 len 8 (pref-sym-algos: 11 7 12 13 2 10 8 9)
    hashed subpkt 21 len 5 (pref-hash-algos: 9 3 8 10 11)
    hashed subpkt 30 len 1 (features: 01)
    subpkt 16 len 8 (issuer key ID 732D8515470359BC)
    data: 28934D133350EF4BA130831027751E1C8D7BEE1B745E185690DE6F3A21CB02CB
    data: 95CC6634A867507CB1CE1D1E6798C417644A8E07ACB4520DB6876D651E80BA9A
# off=283 ctb=9c tag=7 hlen=2 plen=123
:secret sub key packet:
    version 4, algo 18, created 1546083299, expires 0
    pkey[0]: 082A8648CE3D030107 nistp256 (1.2.840.10045.3.1.7)
    pkey[1]: 0452CDCBBF92C1F05B40EAD4C4AD12E967AE6B1BBF457834F75DC61C49FFD29B8AB53365DA4F02D05AAE3B99970521DD3750560A9CDB9F3A6F4CD67DAEBC58FA19
    pkey[2]: 03010807
    skey[3]: 78CFB85E037074B9A6FB14B4A090487EB054F13527B4DD96398B8354F9960EB4
    checksum: 11bf
    keyid: A99B151C882FF69C
# off=408 ctb=88 tag=2 hlen=2 plen=123
:signature packet: algo 19, keyid 732D8515470359BC
    version 4, created 1546083299, md5len 0, sigclass 0x18
    digest algo 10, begin of digest a4 8a
    hashed subpkt 2 len 4 (sig created 2018-12-29)
    hashed subpkt 27 len 1 (key flags: 0C)
    hashed subpkt 22 len 4 (pref-zip-algos: 3 1 0 2)
    hashed subpkt 11 len 8 (pref-sym-algos: 11 7 12 13 2 10 8 9)
    hashed subpkt 21 len 5 (pref-hash-algos: 9 3 8 10 11)
    hashed subpkt 30 len 1 (features: 01)
    subpkt 16 len 8 (issuer key ID 732D8515470359BC)
    data: 7631A142BC418E3C51835283EB9FA2A8FFB14F0B5A94367E04EC2B887567F544
    data: B5209ED7A03825676334C8832239D3A99D4481A2BAAA57A2CC76454B6C75C9DD

The message that fails to decrypt with " gpg: decryption failed: No secret key":

cat cmd_002_log-DecryptCommand/to_gpg_stdin.log | gpg --list-packets --verbose
gpg: public key is 0xA99B151C882FF69C
gpg: encrypted with ECDH key, ID 0xA99B151C882FF69C
gpg: decryption failed: No secret key
# off=0 ctb=84 tag=1 hlen=2 plen=126
:pubkey enc packet: version 3, algo 18, keyid A99B151C882FF69C
    data: 0462E9F40AC8D473C129FFECD9DA3C3621502CE58C2860EC44FB7A078461052F8F5218570A13DFA719412CD0FA88238BA3836363DE70F28E7E8B020C5A58BEE384
    data: 30B8F38183CF85419854384D6BEFEA77E0D69997BD3175E1557CE7352431092611B6C7713F01D0694C7043AAFDB6FD5E2E
# off=128 ctb=d2 tag=18 hlen=3 plen=295 new-ctb
:encrypted data packet:
    length: 295
    mdc_method: 2

What puzzles me is, that the key 0xA99B151C882FF69C is part of the imported private key:

...
:secret sub key packet:
    version 4, algo 18, created 1546083299, expires 0
    pkey[0]: 082A8648CE3D030107 nistp256 (1.2.840.10045.3.1.7)
        [...]
    keyid: A99B151C882FF69C

neuhalje commented 5 years ago

gpg --homedir=$PWD -K
/var/folders/12/vzps3jhs0pz8z_phldh9pz180000gn/T/bouncygpg10153852937528431865/pubring.kbx
------------------------------------------------------------------------------------------
sec   nistp256 2018-12-29 [SCA]
      ADDCFC840DC7196DD1583F26732D8515470359BC
uid           [ unknown] Juliet Capulet <juliet@example.com>
ssb   nistp256 2018-12-29 [E]

neuhalje commented 5 years ago

@vanitasvitae : This very likely also affects pgpainless. Do you have any idea?

vanitasvitae commented 5 years ago

Hm, I think I haven't come across this one yet. I'll check, if PGPainless is also affected later :)

neuhalje commented 5 years ago

RSA (works):

gpg --homedir=$PWD -K
------------------------------------------------------------------------------------------
sec   rsa3072 2018-12-29 [SCEA]
      897A7BE5E76E3311F93C4B946CF5A46E7D732A01
uid           [ unknown] Juliet Capulet <juliet@example.com>

ECC (does not work):

 gpg --homedir=$PWD -K
------------------------------------------------------------------------------------------
sec   nistp256 2018-12-29 [SCA]
      ADDCFC840DC7196DD1583F26732D8515470359BC
uid           [ unknown] Juliet Capulet <juliet@example.com>
ssb   nistp256 2018-12-29 [E]

Manually created key (DSA/ElGamal):

sec   dsa2048/0xB4F93F1AEC8A2DB1 2018-12-30 [SC]
      2533280236B8D4779E948958B4F93F1AEC8A2DB1
uid                   [ultimate] ECC Test (gpg --full-generate-key w. DSA and Elgamal) <ecc-test@example.com>
ssb   elg2048/0x289ED79E07FACF2F 2018-12-30 [E]

Manually created key (nistp256):

sec   nistp256 2018-12-30 [SC]
      36F697C93D20EF1148E38A841290D642DB874F4D
uid           [ unknown] NIST P-256 (gpg --expert --full-gen-key and ECC and ECC) <P-256@example.com>
ssb   nistp256 2018-12-30 [E]

RSA has the keyflags SCEA on the master key but no subkey
ECC has the keyflags SCA (no "E") on the master key and a subkey with "E"
Manually ECC key has the keyflags SC on the master key and a subkey with "E"

vanitasvitae commented 5 years ago

I'm really puzzled about this... This is the third time I try to compose this post, but everytime I do, another error happens/doesn't happen :D

I think PGPainless is affected by this somehow. When I try to recreate this, I get an error like the one you stated above about half the time. The other runs are fine (now I cannot recreate the bug at all). Have you run your test repeatedly?

Here is a key I generated with pgpainless, that seems to work fine for me (although it failed when I imported it in GPG the first time? :thinking:):

An encrypted message:

Which version of bouncycastle are you using? There was a bug that caused exported keys to have a broken format in BC 1.59.

neuhalje commented 5 years ago

Reproducible

The integration test (git clone https://github.com/neuhalje/bouncy-gpg.git ; ./gradlew integrationTest --info; log in build/reports/tests/integrationTest/index.html ) fails consistently. Even when I change the key generation (RSA with subkeys) the result stays the same (RSA works, ECC fails).

BC version

I use bouncy castle 1.60 AND your repairSubkeyPackets.

repairSubkeyPackets warnings

I took your repairSubkeyPackets code. If I understood you correctly, it should not be needed in BC 1.60.

code
called here

The correct key formt according to repairSubkeyPackets is even tested for in the tests. With BC 1.60 these tests fail, when I remove the call to repairSubkeyPackets -- which means that either the bug in BC is not fixed, or your fix did not work (or I borked it ;-)). Removing the fix from the code by commenting out here still fails the integration test.

Your key

The key works without errors:

gpg --homedir=$PWD --import < <(echo "-----BEGIN PGP PRIVATE KEY BLOCK-----
Version: BCPG v1.60

lHcEXCym1BMIKoZIzj0DAQcCAwSOUnSpDZtRwQLp40TbE/OZR7YsW+fHH12pXd+C
/jWhalYHsapuSJAEk5ninvUj9PZK2vyGe9fk1BzoTjAj5AlMAAEAsO61WtpPu7ES
bGoRc64eztRrO71w2+QMJaEtxZXafCgQhrQPZWNrZXlAZGVsZXRlLm1liHUEExMK
AB0FAlwsptQCGyMFFgIDAQAECwkIBwUVCgkICwIeAQAKCRA5gj9k16KVhWUUAP9r
Or9v3MR40UlSjLeg54mRCdys3qdt+Li2KvAlEZJsZwEAhnu7pjw9wJD/SuD9u9iz
P7emL5NtQaAI4OWSx8Cmi/icewRcLKbUEggqhkjOPQMBBwIDBGndg3Z+VjWMMEgo
oo5+OoIfTFJlH5CPcoFXSy/nl6MyauDpxtJlw3hVZB5HJszdheCEe7puCEb+QpHs
FAAErPgDAQgHAAD/YNI+J0K0lUVUfPnZ7rtNUSo0ozZI/+3bJ7XcuYXuqqMSxoh1
BBgTCgAdBQJcLKbUAhsMBRYCAwEABAsJCAcFFQoJCAsCHgEACgkQOYI/ZNeilYUv
TgEA7pvD0K/coWT/UtdnhejRZAEZOMUN4kmDQlfbNY3g9d0A/jDeNTW1BMuqs+LV
JfCK1w+pMIf42/aHpqp3ocZl08TV
=5v4x
-----END PGP PRIVATE KEY BLOCK-----")
gpg: WARNING: unsafe permissions on homedir '/Users/jens/tmp/fff'
gpg: /Users/jens/tmp/fff/trustdb.gpg: trustdb created
gpg: key 39823F64D7A29585: public key "eckey@delete.me" imported
gpg: key 39823F64D7A29585: secret key imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

and

echo '-----BEGIN PGP MESSAGE-----
Version: BCPG v1.60

hH4D+gpfGcqrVhgSAgMEsHr7sSBVTQRWwLuLjM3tv7j5abOIfyu7isx4b+8/ZjQO
K7WLiCZLghp8FO1uFhzxOWaVoQMlGz3qzjoThOfqqjDM8tb2P8XSFo8GINIJ6tVP
QKcoqxCPWApX2Iw5ksR4A9p+Regeg9xIAgRFngxKIEbSZgEQ/V+nrBJr71RKzQ7B
uLXJgDBCNMu4RjGl+yBZM1kjB1J6exr6rc9b43ldnOoh9xaZcflUODJNWFdfcb4z
1woaBjsAeCn4+rvcmcrMK1R8bx6dUK/6xUoXCi+w3TYxU++kJkKTEQ==
=Nlqf
-----END PGP MESSAGE-----' | gpg --homedir=$PWD -d
gpg: WARNING: unsafe permissions on homedir '/Users/jens/tmp/fff'
gpg: encrypted with 256-bit ECDH key, ID FA0A5F19CAAB5618, created 2019-01-02
      "eckey@delete.me"
gpg: Note: sender requested "for-your-eyes-only"
This is a test. Please be readable in GPG!

vanitasvitae commented 5 years ago

Note: The bug I pointed out is not related to https://github.com/bcgit/bc-java/pull/349 but a different bug, which was merged after BC 60 was released.

neuhalje commented 5 years ago

@vanitasvitae: thanks for the clarification

Edit: The error happens with ECC keys, protected with a passphrase or not. Both tests with ECC keys fail.

neuhalje / bouncy-gpg