Targets metadata expiring: no longer have original signing key only delegation key

[dnwake]

Our setup is:

• We use several Jenkins machines to build Docker images
• We use Notary to sign their digests
• In order to allow all the Jenkins machines to push new versions of all the images, we supply an identical delegation keypair to all the machines, and add a delegation for targets/releases to that key whenever we create a new image
• We are now seeing that the targets metadata is expiring / about to expire. It appears to have had a default expiration length of 3 years, but we didn't realize this.
• In some cases, we no longer have the original unique signing keys for these repos
• We have tried to re-add the delegation, but we get an error message "fatal: could not find necessary signing keys, at least one of these keys must be available: XXXXXX"

We need some way to either

• Ideally: extend or eliminate the expiration time on the metadata and delegations
• As a workaround: delete the repo from Notary altogether and recreate it with a much longer expiration date for metadata / delegations

Is it possible to do either of these things without the original signing keys?

Thanks!

Command line output with debugging enabled:

### Trust pinning setup, in case this is relevant
bash-4.1$ cat ~/.notary/config.json
{ "trust_pinning": { "disable_tofu": "true","ca": { "":"/home/jenkins/docker_certs/trust_pinning.crt" } } }

### Attempt to re-add delegation to common delegation keypair
bash-4.1$  /usr/bin/notary -D  -s ${NOTARY_URL} -d /home/jenkins/.docker/trust delegation add ${DOCKER_IMAGE} targets/releases /home/jenkins/docker_certs/delegation.crt
DEBU Using the following trust directory: /home/jenkins/.docker/trust
DEBU Failed to initialize PKCS11 environment: no library found
DEBU Making dir path: /home/jenkins/.docker/trust/tuf/${DOCKER_IMAGE}/changelist
DEBU Adding delegation "targets/releases" with threshold 1, and 1 keys\n
Addition of delegation role targets/releases with keys [f8b24ab3f3e2cc88ed950dca76ba1e09c146a0832566c5134f142825ab24cfcd], to repository "${DOCKER_IMAGE}" staged for next publish.

bash-4.1$  /usr/bin/notary -D  -s ${NOTARY_URL} -d /home/jenkins/.docker/trust publish ${DOCKER_IMAGE}
DEBU Using the following trust directory: /home/jenkins/.docker/trust
Pushing changes to ${DOCKER_IMAGE}
DEBU Failed to initialize PKCS11 environment: no library found
DEBU Making dir path: /home/jenkins/.docker/trust/tuf/${DOCKER_IMAGE}/changelist
DEBU entered ValidateRoot with dns: ${DOCKER_IMAGE}
DEBU found the following root keys: [bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822]
DEBU found 1 valid leaf certificates for ${DOCKER_IMAGE}: bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822
DEBU found no currently valid root certificates for \${DOCKER_IMAGE}, using trust_pinning config to bootstrap trust
DEBU  role has key IDs: bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822
DEBU verifying signature for key ID: bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822
DEBU Root validation succeeded for ${DOCKER_IMAGE}
DEBU entered ValidateRoot with dns: ${DOCKER_IMAGE}
DEBU found the following root keys: [bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822]
DEBU found 1 valid leaf certificates for ${DOCKER_IMAGE}: bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822
DEBU found no currently valid root certificates for ${DOCKER_IMAGE}, using trust_pinning config to bootstrap trust
DEBU  role has key IDs: bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822
DEBU verifying signature for key ID: bbd24fde73ca0a6490b1b19ffbd4ba426fcd889040d94b0b1f5d6e9e2d796822
DEBU Root validation succeeded for ${DOCKER_IMAGE}
DEBU 200 when retrieving metadata for root
DEBU updating TUF client
DEBU Loading timestamp...
DEBU 200 when retrieving metadata for timestamp
DEBU timestamp role has key IDs: f0a0d9a40f84cc9bb1169bffd11b44bfe446ceb6e1d218d28990a829f1815286
DEBU verifying signature for key ID: f0a0d9a40f84cc9bb1169bffd11b44bfe446ceb6e1d218d28990a829f1815286
DEBU timestamp role has key IDs: f0a0d9a40f84cc9bb1169bffd11b44bfe446ceb6e1d218d28990a829f1815286
DEBU verifying signature for key ID: f0a0d9a40f84cc9bb1169bffd11b44bfe446ceb6e1d218d28990a829f1815286
DEBU successfully verified downloaded timestamp
DEBU Loading snapshot...
DEBU snapshot role has key IDs: 2b61cb0636ef5ff52a0b80d78ba87356cc871c2fa165b7886ca630ebe2045edb
DEBU verifying signature for key ID: 2b61cb0636ef5ff52a0b80d78ba87356cc871c2fa165b7886ca630ebe2045edb
DEBU successfully verified cached snapshot
DEBU Loading targets...
DEBU targets role has key IDs: 9f124590302353df92642980ebc20e4147cb2a0941f7de5e3347142001bc03b0
DEBU verifying signature for key ID: 9f124590302353df92642980ebc20e4147cb2a0941f7de5e3347142001bc03b0
DEBU successfully verified cached targets
DEBU Loading targets/releases...
DEBU targets/releases role has key IDs: 1630fbb1e4e5c9bc4fc77e559c7e0f6de8dfda9bf621f2c54150f522dcabeead
DEBU verifying signature for key ID: 1630fbb1e4e5c9bc4fc77e559c7e0f6de8dfda9bf621f2c54150f522dcabeead
DEBU successfully verified cached targets/releases
WARN targets metadata is nearing expiry, you should re-sign the role metadata
DEBU Failed to initialize PKCS11 environment: no library found
DEBU Failed to initialize PKCS11 environment: no library found
DEBU Error applying changelist
* fatal: could not find necessary signing keys, at least one of these keys must be available: 9f124590302353df92642980ebc20e4147cb2a0941f7de5e3347142001bc03b0

### LIST OF UPCOMING EXPIRATION DATES
bash-4.1$ find .docker/trust/ -type f  | grep j${DOCKER_IMAGE} | sort | xargs grep -oP "expires...[^\"]+"
.docker/trust/tuf/${DOCKER_IMAGE}/metadata/root.json:expires":"2026-09-26T15:27:30.887590816-07:00
.docker/trust/tuf/${DOCKER_IMAGE}/metadata/snapshot.json:expires":"2022-06-04T03:26:12.084278292Z
.docker/trust/tuf/${DOCKER_IMAGE}/metadata/targets.json:expires":"2019-09-28T15:27:31.231893787-07:00
.docker/trust/tuf/${DOCKER_IMAGE}/metadata/targets/releases.json:expires":"2022-06-03T20:26:12.030907135-07:00
.docker/trust/tuf/${DOCKER_IMAGE/metadata/timestamp.json:expires":"2019-10-09T01:50:13.211124782Z

### CONTENT OF TARGETS.JSON
bash-4.1$ cat .docker/trust/tuf/${DOCKER_IMAGE}/metadata/targets.json | jq -C -M .
{
  "signed": {
    "_type": "Targets",
    "delegations": {
      "keys": {
        "1630fbb1e4e5c9bc4fc77e559c7e0f6de8dfda9bf621f2c54150f522dcabeead": {
          "keytype": "rsa-x509",
          "keyval": {
            "private": null,
            "public": "XXXXXXXXXXXXX"
          }
       }
      },
      "roles": [
        {
          "keyids": [
            "1630fbb1e4e5c9bc4fc77e559c7e0f6de8dfda9bf621f2c54150f522dcabeead"
          ],
          "name": "targets/releases",
          "paths": [
            ""
          ],
          "threshold": 1
        }
      ]
    },
    "expires": "2019-09-28T15:27:31.231893787-07:00",
    "targets": {},
    "version": 3
  },
  "signatures": [
    {
      "keyid": "9f124590302353df92642980ebc20e4147cb2a0941f7de5e3347142001bc03b0",
      "method": "ecdsa",
      "sig": "XXXXXXXX"
    }
  ]
}

[MohamedTalhaoui]

Hi @dnwake can you please share more details on your setup and how you distribute keys to your jenkins machines. I have a similar use case where I have multiple clients (not jenkins) that need to push and sign different tags of the same image. I have tried to configure trust_pinning and disable_tofu but nothing works. I am still seeing that first client will generate target keys for the GUN image and other clients are rejected because of missing the target key. I would be grateful if you can share your knowledge here.

[dnwake]

@MohamedTalhaoui

We distribute two keypairs (a delegation keypair and a trust-pinning keypair) to all our Jenkins machines via Puppet.

When creating a new GUN on one of the Jenkins machines:

• Generate a fresh root keypair on the machine. Generate a cert from the public key.
• Add the GUN locally using the freshly-generated root keypair (we had to modify the Notary client to allow us to specify both the private key and the cert).
• Add a delegation to the delegation cert, to allow any of the Jenkins machine to submit subsequent images to the GUN
• Publish to the Notary repository.

The trust-pinning keypair is used to stop the TOFU loophole.

The problem described in this thread arose because we discarded the generated root keypairs, the delegation keypairs eventually expired, and because we no longer had the root keypairs we weren't able to extend / renew the delegation.

[MohamedTalhaoui]

Hi @dnwake

thanks a lot for your reply. Could you also please share how these steps translates to notary/docker trust commands ?

My setup is based on Harbor deployed on a K8S cluster. I have deployed some pods running a docker daemon and having notary cli installed. These pods are created and destroyed dynamically to sign images. The root and delegate keys are mounted to the pods. Each time a pod is created, it does the following:

# login to docker registry
docker login -u $USERNAME -p $PASSWORD $HARBOR"
# load root key
docker trust key load signer-root.key --name root
# load delegate key
docker trust key load signer-delegate.key --name signer
# add delegate, this command will fail unless the pod has initiated the GUN
docker trust signer add --key signer-delegate.crt signer $HARBOR/$DST_REPO
# sign image
DOCKER_CONTENT_TRUST=1 docker push $HARBOR/$DST_REPO:$DST_TAG"

So far it is woking but I am facing the following limitations:

• Cannot get rid of TOFU even with trust_pinning
• For a specific GUN, first pod only (the one initiating the trust collection) will have the repository key. So when this pod is killed, keys are lost and subsequent actions on the repo are rejected from other pods.

Is there a way to provide the keys to use for GUNs. I was expecting it was possible with trust_pinning but I am either wrong or missing something.

Any help is welcome.