stock gpg cannot import public key Perkeep generates

[epaulson]

This is a followup to the discussion here: https://groups.google.com/forum/#!topic/camlistore/Q5Ljhehrzg0

tl;dr: camlistore doesn't put enough info into the public key it serializes for a stock openpgp client to use it. Adding the extra bits changes the sha-1 of the blob, so an upgrade path needs to be considered for a fix.

Using this:

zebra:test cpaulser$ gpg --version
gpg (GnuPG/MacGPG2) 2.0.27
libgcrypt 1.6.3
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Pubkey: RSA, RSA, RSA, ELG, DSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

I am unable to import the public keys camlistore stores. With a fresh build and simply running bin/camlistored, I get a public key like so:

zebra:test cpaulser$ cat public-key-blob.stock 
-----BEGIN PGP PUBLIC KEY BLOCK-----

xsBNBFV0plkBCACr1BZ5JzrQL23g97GaOxALEcAMzYxYvuwSHlkR1CCgoRklcPGo
DuihmVpDzC1/AeX1aUINhMndOuP0ip9cAHm4WGGL+RbUtXuSkK2Z6CHmec8cPyXX
4mu3t4D5ZfrF0Q9TamVjmoTwseczXjmJIsPhPqw4aWhkCDxYGgaVm68SIdjzbhp2
X8mM4bLtF9FD5CMHqRI2aL24CcI7JikYEopHqIh5QhGzU1ak8JX880GD3vPw7jN0
fEFaMmi9mUzYxL8hpvfSoUfz55iMwmb/lZ320kYF16Tsqr3VV+kr0A2uhnZ3Nd6u
nLILaWA0oEnpyQqVNy/UE8bHqgFtO6clEaJlABEBAAE=
=XHHk
-----END PGP PUBLIC KEY BLOCK-----

Using gpg to import it fails:

zebra:test cpaulser$ gpg --no-default-keyring --keyring ./stock.gpg --import public-key-blob.stock 
gpg: keyring `./stock.gpg' created
gpg: key 28FD6F75: no user ID
gpg: Total number processed: 1
zebra:test cpaulser$ ls -l
total 8
-rw-r--r--  1 cpaulser  staff  449 Jun  7 15:16 public-key-blob.stock
-rw-------  1 cpaulser  staff    0 Jun  7 15:16 stock.gpg

The trick seems to be to add in a UID and Self-signing packet. As-is now, when Camlistore serializes the public key, it only serializes the public key, and not any of the other entity metadata it extracts from identity-secring.gpg, ie in jsonsign/keys.go ArmoredPublicKey() it only does entity.PrivateKey.PublicKey.Serialize(wc). I think that should either be https://godoc.org/golang.org/x/crypto/openpgp#Entity.Serialize or at least follow its algorithm (though it might be way more than is really needed - but at the same time, if you're using an existing keypair, it'd be good to also publish any web of trust signatures).

This was the minimal diff I used, but Entity.Serialize is probably better:

zebra:camlistore cpaulser$ git diff
diff --git a/pkg/jsonsign/keys.go b/pkg/jsonsign/keys.go
index 14a2ae4..4114f19 100644
--- a/pkg/jsonsign/keys.go
+++ b/pkg/jsonsign/keys.go
@@ -83,15 +83,23 @@ func openArmoredPublicKeyFile(reader io.ReadCloser) (*packet.PublicKey, error) {
        if block.Type != "PGP PUBLIC KEY BLOCK" {
                return nil, errors.New("Invalid public key blob.")
        }
-       p, err := packet.Read(block.Body)
+        var opr = packet.NewOpaqueReader(block.Body)
+        op, err := opr.Next()
        if err != nil {
                return nil, fmt.Errorf("Invalid public key blob: %v", err)
        }
-
+        p, err := op.Parse()
+        if err != nil {
+                return nil, fmt.Errorf("Could not parse even a single packet: %v", err)
+        }
        pk, ok := p.(*packet.PublicKey)
        if !ok {
                return nil, fmt.Errorf("Invalid public key blob; not a public key packet")
        }
+        // burn off any remaining packets
+        for err == nil {
+                _, err = opr.Next()
+        }
        return pk, nil
 }

@@ -149,6 +157,16 @@ func ArmoredPublicKey(entity *openpgp.Entity) (string, error) {
        if err != nil {
                return "", err
        }
+        for _, ident := range entity.Identities {
+               err = ident.UserId.Serialize(wc)
+               if err != nil {
+                       return "", err
+               }
+               err = ident.SelfSignature.Serialize(wc)
+               if err != nil {
+                       return "", err
+               }
+        }
        wc.Close()
        if !bytes.HasSuffix(buf.Bytes(), newlineBytes) {
                buf.WriteString("\n")

If I leave off the ident.SelfSignature.Serialize bit, I get:

zebra:test cpaulser$ cat public-key-blob.withuid
-----BEGIN PGP PUBLIC KEY BLOCK-----

xsBNBFV0plkBCACr1BZ5JzrQL23g97GaOxALEcAMzYxYvuwSHlkR1CCgoRklcPGo
DuihmVpDzC1/AeX1aUINhMndOuP0ip9cAHm4WGGL+RbUtXuSkK2Z6CHmec8cPyXX
4mu3t4D5ZfrF0Q9TamVjmoTwseczXjmJIsPhPqw4aWhkCDxYGgaVm68SIdjzbhp2
X8mM4bLtF9FD5CMHqRI2aL24CcI7JikYEopHqIh5QhGzU1ak8JX880GD3vPw7jN0
fEFaMmi9mUzYxL8hpvfSoUfz55iMwmb/lZ320kYF16Tsqr3VV+kr0A2uhnZ3Nd6u
nLILaWA0oEnpyQqVNy/UE8bHqgFtO6clEaJlABEBAAHNDChjYW1saXN0b3JlKQ==
=PRX+
-----END PGP PUBLIC KEY BLOCK-----
zebra:test cpaulser$ gpg --no-default-keyring --keyring ./stock.gpg --import public-key-blob.withuid 
gpg: key 28FD6F75: no valid user IDs
gpg: this may be caused by a missing self-signature
gpg: Total number processed: 1
gpg:           w/o user IDs: 1
zebra:test cpaulser$ ls -l
total 16
-rw-r--r--  1 cpaulser  staff  449 Jun  7 15:16 public-key-blob.stock
-rw-r--r--  1 cpaulser  staff  469 Jun  7 15:24 public-key-blob.withuid
-rw-------  1 cpaulser  staff    0 Jun  7 15:16 stock.gpg

Adding in the SelfSignature succeeds:

zebra:test cpaulser$ cat public-key-blob.withuidselfsig
-----BEGIN PGP PUBLIC KEY BLOCK-----

xsBNBFV0plkBCACr1BZ5JzrQL23g97GaOxALEcAMzYxYvuwSHlkR1CCgoRklcPGo
DuihmVpDzC1/AeX1aUINhMndOuP0ip9cAHm4WGGL+RbUtXuSkK2Z6CHmec8cPyXX
4mu3t4D5ZfrF0Q9TamVjmoTwseczXjmJIsPhPqw4aWhkCDxYGgaVm68SIdjzbhp2
X8mM4bLtF9FD5CMHqRI2aL24CcI7JikYEopHqIh5QhGzU1ak8JX880GD3vPw7jN0
fEFaMmi9mUzYxL8hpvfSoUfz55iMwmb/lZ320kYF16Tsqr3VV+kr0A2uhnZ3Nd6u
nLILaWA0oEnpyQqVNy/UE8bHqgFtO6clEaJlABEBAAHNDChjYW1saXN0b3JlKcLA
YgQTAQgAFgUCVXSmWQkQXP79aCj9b3UCGwMCGQEAAI9pCABsCqdF3l3cxz+lqhf9
fZgy7cuMAp/P+CAAm8mAjYBb8YMbv6Xb3Kac1UDlQ1/n9hAEX0Qx4Llb8mp65wUS
xXtO4fVjB3tmHVPCsz8axTPkrp2cCFnh1x8fxtTLx4wTj6K/R5nXXB03advn2o6V
LL5CeJVNcbHwhl5JTOGuuaJlqeN2++YlhANLYM1ahjlSnP58I29+SXTPcMOIivkZ
RylA6LfLGPRszNTdHclsM1d8qiauMra1u6+jQJSBh2v+g1hhSHq5v4gQj6Lu8nrU
Ig2ipWum4Eo/h21k4S+wLZ7ZzoHcEr65XRjBvklA7icZP+KmXjNQ+UnG5MQsio5I
B50K
=VE2I
-----END PGP PUBLIC KEY BLOCK-----
zebra:test cpaulser$ gpg --no-default-keyring --keyring ./stock.gpg --import public-key-blob.withuidselfsig 
gpg: key 28FD6F75: public key "(camlistore)" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
zebra:test cpaulser$ ls -l
total 32
-rw-r--r--  1 cpaulser  staff  449 Jun  7 15:16 public-key-blob.stock
-rw-r--r--  1 cpaulser  staff  469 Jun  7 15:24 public-key-blob.withuid
-rw-r--r--  1 cpaulser  staff  864 Jun  7 15:26 public-key-blob.withuidselfsig
-rw-------  1 cpaulser  staff  583 Jun  7 15:27 stock.gpg
-rw-------  1 cpaulser  staff    0 Jun  7 15:16 stock.gpg~

If you simply serialize extra information, Camlistore panics on startup in pkg/schema/sign.go: it still only reads the public key packet from the blob in openArmoredPublicKeyFile, and as a sanity check computes the hash of what it read to be sure that the blob matches. Now that the blob has additional OpenPGP packets, the hash of the partial read camli does doesn't match the blob.

This turned out to be a pretty easy fix, because a TeeReader is used in pkg/shema/sign.go to copy bytes into the buffer to hash without ParseArmoredPublicKey in jsonsign having to know anything about it. There's also a great API in the crypto/openpgp package that let you read packets and just ignore what you're not interested in, so I slid that in, read and gave back the public key packet to the callers, and just did the extra reads in the background to make sure the hashes match without needing to change any callers. A real fix might want to make sure that all callers are aware that the hash of the key blobfile does not match the hash of the bytes openArmoredPublicKeyFile returns.

The other challenging issue is that existing signed blobs reference the hash of the signing key - which is generated on startup from ~/.config/camlistore/identity-secring.gpg. With the changes above, the key will change, and existing deployments will have blobrefs to a blob that is no longer published. If you decide to publish additional data in the public key, I imagine you'll want to publish the public key using both the old serialization scheme and a new one.

[epaulson]

It turns out that gpg can handle a key missing a self-signature at the command line, with the '--allow-non-selfsigned-uid' commandline option.

https://www.gnupg.org/gph/en/manual/r1424.html

zebra:test cpaulser$ gpg --no-default-keyring --allow-non-selfsigned-uid --keyring ./stock.gpg --import public-key-blob.withuid
gpg: keyring `./stock.gpg' created
gpg: key 28FD6F75: accepted non self-signed user ID "(camlistore)"
gpg: key 28FD6F75: public key "(camlistore)" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
zebra:test cpaulser$ ls -l stock.gpg
-rw-------  1 cpaulser  staff  286 Jun 23 15:17 stock.gpg

gpg still chokes on the default camlistore public key because there's no UID data embedded in the key. There is a '--local-user' commandline option that I thought might create a UID as part of the import, but I didn't have any luck with it.

zebra:test cpaulser$ rm stock.gpg
zebra:test cpaulser$ gpg --no-default-keyring --allow-non-selfsigned-uid --local-user camlistore@camlistore.com --keyring ./stock.gpg --import public-key-blob.stock
gpg: keyring `./stock.gpg' created
gpg: key 28FD6F75: no user ID
gpg: Total number processed: 1
zebra:test cpaulser$ ls -l stock.gpg
-rw-------  1 cpaulser  staff  0 Jun 23 15:19 stock.gpg
zebra:test cpaulser$

The gpg manual says of --allow-non-selfsigned-uid: "You should really avoid using it, because OpenPGP has better mechanics to do separate signing and encryption keys." but I'm still searching for the right recipe to use the camlistore-generated public key with the gpg commandline tools.

[aviau]

@epaulson Hey! I'd like to get that fixed. I am working on a python implementation of some perkeep internals and this makes things way more complicated that they need to be.

The other challenging issue is that existing signed blobs reference the hash of the signing key - which is generated on startup from ~/.config/camlistore/identity-secring.gpg. With the changes above, the key will change, and existing deployments will have blobrefs to a blob that is no longer published.

I don't understand what you mean by that. Why would the blobref no longer be published? Isn't it stored permanently just like all other blobs?

Edit: ah, you probably refer to the signhandler endpoints where we wouldn't advertise our older key anymore?