Update of nearing expiry delegation key with validity extended key

[ase-101]

I recently noticed "nearing expiry" warning when pulling image from one of the trusted docker repository. Delegation process was used to sign the images.

i tried the below approach

1. extended the validity of collaborator's public key.
2. Removed the delegation from docker repository.
3. Added back delegation with collaborator's public key with extended validity.

Now when i try to list the delegation i see this warning WARN[0002] Error getting targets/releases: valid signatures did not meet threshold for targets/releases

and signing also fails with below error * fatal: trust server rejected operation.

Is it required to generate a new key pair as collaborator a redo all the steps and resign all the images ?

I am sorry if this is not the correct window. thank you.

[tim-gp]

I have seen the same issue when using delegation keys.

@AnushaSunkada would you mind renaming the issue "Revoking GUN delegation keys for a role prevents further trust operations due to threshold"

Here are the steps using the notary compose file (from master) and a locally-running registry:

# start notary and a registry
docker-compose up -d
docker run -d -p 5000:5000 --restart=always --name registry registry:2

# create a root key, initialise GUN and rotate snapshot key
notary key generate
notary -s https://localhost:4443 --tlscacert fixtures/root-ca.crt init localhost:5000/busybox -p 
notary -s https://localhost:4443 --tlscacert fixtures/root-ca.crt key rotate localhost:5000/busybox snapshot -r 

# generate two private and public keys, chmod private keys to 600 so docker can load
openssl genrsa -out delegate1.key 2048
openssl rsa -pubout -in delegate1.key -out delegate1.crt
openssl genrsa -out delegate2.key 2048
openssl rsa -pubout -in delegate2.key -out delegate2.crt
chmod 600 delegate*.key

# add the first delegate key
notary -s https://localhost:4443 --tlscacert fixtures/root-ca.crt delegation add localhost:5000/busybox targets/releases delegate1.crt --all-paths -p

# setup DCT
docker pull busybox
docker tag busybox localhost:5000/busybox:latest
export DOCKER_CONTENT_TRUST=1
export DOCKER_CONTENT_TRUST_SERVER=https://localhost:4443
docker push localhost:5000/busybox:latest # fails 'no valid signing keys for delegation roles'

# load first delegate key into docker
docker trust key load delegate1.key
docker push localhost:5000/busybox:latest # succeeds

# add second delegate key to notary
notary -s https://localhost:4443 --tlscacert fixtures/root-ca.crt delegation add localhost:5000/busybox targets/releases delegate2.crt --all-paths -p

# sign with first key
docker push localhost:5000/busybox:latest # succeeds

# remove first key from docker and load second key
rm ~/.docker/trust/private/*
docker trust key load delegate2.key

# sign with second key
docker push localhost:5000/busybox:latest # succeeds

# remove first key from notary, then list delegations
notary -s https://localhost:4443 --tlscacert fixtures/root-ca.crt delegation purge localhost:5000/busybox --key <firstkeyid> -p
notary -s https://localhost:4443 --tlscacert fixtures/root-ca.crt delegation list localhost:5000/busybox # shows WARN[0000] Error getting targets/releases: valid signatures did not meet threshold for targets/releases

docker push localhost:5000/busybox:latest # fails 'WARN[0000] Error getting targets/releases: valid signatures did not meet threshold for targets/releases - failed to sign localhost:5000/busybox:latest: trust server rejected operation.'

Here is the debug output from delegation list after purging the first key:

notary -D -s https://localhost:4443 --tlscacert fixtures/root-ca.crt delegation list localhost:5000/busybox
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /Users/timgp/.notary 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /Users/timgp/.notary/tuf/localhost:5000/busybox/changelist 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/busybox 
DEBU[0000] found the following root keys: [ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/busybox: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/busybox 
DEBU[0000] checking root against trust_pinning config for localhost:5000/busybox 
DEBU[0000] checking trust-pinning for cert: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000]  role has key IDs: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000] verifying signature for key ID: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000] root validation succeeded for localhost:5000/busybox 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/busybox 
DEBU[0000] found the following root keys: [ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/busybox: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/busybox 
DEBU[0000] checking root against trust_pinning config for localhost:5000/busybox 
DEBU[0000] checking trust-pinning for cert: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000]  role has key IDs: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000] verifying signature for key ID: ee39a8b99440b38fc3df04ec51bac125bbab442f4498b84bb1a2f64a8c49c741 
DEBU[0000] root validation succeeded for localhost:5000/busybox 
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 6b959d30512cabb4e70fcb0cf2a66db478c0875b7b6a3aa50974eeb669848ae4 
DEBU[0000] verifying signature for key ID: 6b959d30512cabb4e70fcb0cf2a66db478c0875b7b6a3aa50974eeb669848ae4 
DEBU[0000] timestamp role has key IDs: 6b959d30512cabb4e70fcb0cf2a66db478c0875b7b6a3aa50974eeb669848ae4 
DEBU[0000] verifying signature for key ID: 6b959d30512cabb4e70fcb0cf2a66db478c0875b7b6a3aa50974eeb669848ae4 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] snapshot role has key IDs: 62bceb20b28ce8b0005a31dda69d54bf61022d9ee53eb7104e45dbd15093004a 
DEBU[0000] verifying signature for key ID: 62bceb20b28ce8b0005a31dda69d54bf61022d9ee53eb7104e45dbd15093004a 
DEBU[0000] successfully verified cached snapshot        
DEBU[0000] Loading targets...                           
DEBU[0000] targets role has key IDs: 71a043b9200cf32c4bb7fa056082b37c8e43fe3869b51c43b12ea81e55abf7b9 
DEBU[0000] verifying signature for key ID: 71a043b9200cf32c4bb7fa056082b37c8e43fe3869b51c43b12ea81e55abf7b9 
DEBU[0000] successfully verified cached targets         
DEBU[0000] Loading targets/releases...                  
DEBU[0000] targets/releases role has key IDs: 363f90a2df620cc9e7b79bed87aa8b66489697c4e312608376f4f8a1abef918d 
DEBU[0000] verifying signature for key ID: cb1f6014718e5c2346f887c60523f38bc4967ce9fe8f1689c70cd08b73a5613b 
DEBU[0000] continuing b/c keyid lookup was nil: cb1f6014718e5c2346f887c60523f38bc4967ce9fe8f1689c70cd08b73a5613b
DEBU[0000] cached targets/releases is invalid (must download): valid signatures did not meet threshold for targets/releases 
DEBU[0000] 200 when retrieving metadata for targets/releases.9691d93dcd84e356822136ec58d5e96ea5b428d5b39a710f044e07be35b94f96 
DEBU[0000] targets/releases role has key IDs: 363f90a2df620cc9e7b79bed87aa8b66489697c4e312608376f4f8a1abef918d 
DEBU[0000] verifying signature for key ID: cb1f6014718e5c2346f887c60523f38bc4967ce9fe8f1689c70cd08b73a5613b 
DEBU[0000] continuing b/c keyid lookup was nil: cb1f6014718e5c2346f887c60523f38bc4967ce9fe8f1689c70cd08b73a5613b

DEBU[0000] downloaded targets/releases.9691d93dcd84e356822136ec58d5e96ea5b428d5b39a710f044e07be35b94f96 is invalid: valid signatures did not meet threshold for targets/releases 
WARN[0000] Error getting targets/releases: valid signatures did not meet threshold for targets/releases 

ROLE                PATHS             KEY IDS                                                             THRESHOLD
----                -----             -------                                                             ---------
targets/releases    "" <all paths>    363f90a2df620cc9e7b79bed87aa8b66489697c4e312608376f4f8a1abef918d    1

The interesting line is DEBU[0000] verifying signature for key ID: cb1f6014718e5c2346f887c60523f38bc4967ce9fe8f1689c70cd08b73a5613b - in my example the first delegate key was id cb1... and the second was id 363.... It appears that the targets/releases role may still be associated with the first delegate key.

When I re-add the first public key with delegation add the threshold messages stop and I can continue to push and sign content.

I would really like to understand how delegation keys can be updated in the case of either certificate expiry or key compromise as I cannot see how it works in practice.

[tim-gp]

FWIW, I tried the irc channel too, but it was empty.

[dnwake]

@AnushaSunkada Did you manage to resolve this issue? I am seeing a very similar problem: https://github.com/theupdateframework/notary/issues/1499

[ghost]

@tim-gp did you figure out the cause behind this? I am hitting the same issue here, this is annoying since this prevent me from removing any old keys from a delegation role. As soon as I remove a key from a delegation, the threshold messages prevent my from signing with new keys, the only way to work around this is to use the new key with notary witness as if the new key is used by a new collaborator who just take over the repo and continue to work in it.

[ase-101]

Although not able to know the cause. Below scripts by my friend can help and save your time.

https://github.com/mandeepdhiman123/SecurityAutomation