search

notaryproject / notary

Notary is a project that allows anyone to have trust over arbitrary collections of data
Apache License 2.0
3.22k stars 508 forks source link

Unable To Publish Images Using Delegation Keys w/ Docker Content Trust - Del. Passphrase Not Accepted #1281

Open unullmass opened 6 years ago

unullmass commented 6 years ago

I've been unable to publish signed images using Docker Content Trust with the following Notary setup.

# notary version

notary Version: 0.5.0 Git commit: a41821f

# docker version

Client: Version: 1.13.1 API version: 1.26 Go version: go1.6.2 Git commit: 092cba3 Built: Thu Nov 2 20:40:23 2017 OS/Arch: linux/amd64

Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Go version: go1.6.2 Git commit: 092cba3 Built: Thu Nov 2 20:40:23 2017 OS/Arch: linux/amd64 Experimental: false

Also tried with a different docker daemon and client:

# docker version

Client: Version: 17.09.1-ce API version: 1.32 Go version: go1.8.3 Git commit: 19e2cf6 Built: Thu Dec 7 22:24:23 2017 OS/Arch: linux/amd64

Server: Version: 17.09.1-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: 19e2cf6 Built: Thu Dec 7 22:23:00 2017 OS/Arch: linux/amd64 Experimental: false

I've set these vars for enabling Docker Content Trust:

export DOCKER_CONTENT_TRUST=1
export DOCKER_CONTENT_TRUST_SERVER=https://localhost:4443

I'm able to push signed images using the targets key: # docker push localhost:5000/alpine:signed

The push refers to a repository [localhost:5000/alpine] 04a094fe844e: Layer already exists signed: digest: sha256:0a0caec02053bbe352451e3422e5d7d1d70874ad7936786370117c3086898e0c size: 528 Signing and pushing trust metadata You are about to create a new root signing key passphrase. This passphrase will be used to protect the most sensitive key in your signing system. Please choose a long, complex passphrase and be careful to keep the password and the key file itself secure and backed up. It is highly recommended that you use a password manager to generate the passphrase and keep it safe. There will be no way to recover this key. You can find the key in your config directory. Enter passphrase for new root key with ID 77d7d70: Repeat passphrase for new root key with ID 77d7d70: Enter passphrase for new repository key with ID 1b80afc (localhost:5000/alpine): Repeat passphrase for new repository key with ID 1b80afc (localhost:5000/alpine): Finished initializing "localhost:5000/alpine" Successfully signed "localhost:5000/alpine":signed

# notary list localhost:5000/alpine

NAME DIGEST SIZE (BYTES) ROLE

signed 0a0caec02053bbe352451e3422e5d7d1d70874ad7936786370117c3086898e0c 528 targets

Now I proceed to generate a keypair for a collaboration role: # openssl req -x509 -newkey rsa:2048 -keyout priv.pem -out pub.pem -nodes -batch

Generating a 2048 bit RSA private key
...................................+++
...................................................+++
writing new private key to 'priv.pem'
-----

# ls

priv.pem pub.pem

# cfssl certinfo -cert pub.pem

{ "subject": { "country": "AU", "organization": "Internet Widgits Pty Ltd", "province": "Some-State", "names": [ "AU", "Some-State", "Internet Widgits Pty Ltd" ] }, "issuer": { "country": "AU", "organization": "Internet Widgits Pty Ltd", "province": "Some-State", "names": [ "AU", "Some-State", "Internet Widgits Pty Ltd" ] }, "serial_number": "14404122303687016242", "not_before": "2017-12-18T19:21:40Z", "not_after": "2018-01-17T19:21:40Z", "sigalg": "SHA256WithRSA", "authority_key_id": "45:FA:70:6B:27:92:DC:63:C7:AD:DB:FC:8A:BF:75:76:98:7D:38:AA", "subject_key_id": "45:FA:70:6B:27:92:DC:63:C7:AD:DB:FC:8A:BF:75:76:98:7D:38:AA", "pem": "-----BEGIN CERTIFICATE-----\nMIIDXTCCAkWgAwIBAgIJAMfluOs8i78yMA0GCSqGSIb3DQEBCwUAMEUxCzAJBgNV\nBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX\naWRnaXRzIFB0eSBMdGQwHhcNMTcxMjE4MTkyMTQwWhcNMTgwMTE3MTkyMTQwWjBF\nMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50\nZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEA5+KzOdglr6YvW490mQCeEEU4+jVZbG3h3G5E2oE4JwZgyRPLnp8DH+Xe\nqIh4SP4ILNFJOVcJgZAq/wtO+lawxkZqQRWG+7gRDElhxlAJthPayNDzwT8xUZe3\nBNajCqvniuep3vetMLdT/9nh+sKCxag02TJIARIgmE75J5P/+eBj6IJ71d7jtanX\ncx06Q6qE/TG0fquSYlqEamZSBu55Z50AgQJDQYquCOo6l/Om6tmE9V97Lvk5q38o\nsbFIUZpRyJ8g5TcDCFk3uUMVwBL0F6v93XqPWYY1pQYSAJrCHBICDq3uJQ/OJSR9\nWsyMgUaPHDko9mjIdYwnUUzecq11UQIDAQABo1AwTjAdBgNVHQ4EFgQURfpwayeS\n3GPHrdv8ir91dph9OKowHwYDVR0jBBgwFoAURfpwayeS3GPHrdv8ir91dph9OKow\nDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAd60eUyfLjyYMushbemk+\nUP2Ls0oB2aMdQL+xMQxbZp1Mx8xlhqqAhPewRlYxh+L5Wbp8w6Fal+AVGcpjsTHq\ngNM6AYCXnfxKfDcsUT2o00GWmRKODwnm5FI3lPuVU30QBR45jatTTd+qT3wGa50Y\nm2J249DGiEYgVXUAb7E8A3w9ZXYsj3DBdvK/MFkM3aFocRvjE6IR11ZLtw1XgpYU\n+be7Cn0I44aoX2zEeUnY5RCWkjKi+NiCbuDJFL+S7bYQZ0jkj2Kvs+eonC4lP06e\np3PjE8qWBAlXK9gM1z7FJ4J0sFrX9KPKk0lKXjHQV+nUEMLZhPYnfXKNUx7hkYVt\nMg==\n-----END CERTIFICATE-----\n" }

# notary delegation add localhost:5000/alpine targets/releases pub.pem --all-paths

Addition of delegation role targets/releases with keys [e55d5bece6e3cbf3a8c05558cbde253f7dd5dc57150b4cdffb53e7da2ff5b12e], with paths ["" ], to repository "localhost:5000/alpine" staged for next publish.

# notary publish localhost:5000/alpine

Pushing changes to localhost:5000/alpine Enter passphrase for targets key with ID 1b80afc: Successfully published changes for repository localhost:5000/alpine

After importing the private key: # notary key import priv.pem -g localhost:5000/alpine -r targets/release

Enter passphrase for new targets/release key with ID e55d5be: Repeat passphrase for new targets/release key with ID e55d5be:

# notary key list

ROLE GUN KEY ID LOCATION

root 77d7d70c6e124c37dabe20c794f9be15878d0e9a16ccf35da3442d32afc647fa /root/.docker/trust/private targets/release e55d5bece6e3cbf3a8c05558cbde253f7dd5dc57150b4cdffb53e7da2ff5b12e /root/.docker/trust/private targets localhost:5000/alpine 1b80afca93cacebb6c485b7f7ce71c6bed2252ce869953ac500e25e2a6bd0230 /root/.docker/trust/private

I attempt a push using the delegation to sign: # docker push localhost:5000/alpine:signed

The push refers to a repository [localhost:5000/alpine] 04a094fe844e: Layer already exists signed: digest: sha256:0a0caec02053bbe352451e3422e5d7d1d70874ad7936786370117c3086898e0c size: 528 Signing and pushing trust metadata WARN[0000] certificate with CN is near expiry Enter passphrase for targets/release key with ID e55d5be: Passphrase incorrect. Please retry. Enter passphrase for targets/release key with ID e55d5be: Passphrase incorrect. Please retry. Enter passphrase for targets/release key with ID e55d5be: Passphrase incorrect. Please retry. Enter passphrase for targets/release key with ID e55d5be: ERRO[0007] couldn't add target to targets/releases: could not find necessary signing keys, at least one of these keys must be available: e55d5bece6e3cbf3a8c05558cbde253f7dd5dc57150b4cdffb53e7da2ff5b12e Failed to sign "localhost:5000/alpine":signed - could not find necessary signing keys, at least one of these keys must be available: e55d5bece6e3cbf3a8c05558cbde253f7dd5dc57150b4cdffb53e7da2ff5b12e Error: could not find signing keys for remote repository localhost:5000/alpine, or could not decrypt signing key: could not find necessary signing keys, at least one of these keys must be available: e55d5bece6e3cbf3a8c05558cbde253f7dd5dc57150b4cdffb53e7da2ff5b12e

Unable to find a workaround for this as of now. Please let me know if I'm missing something. All steps follow the Notary documentation here.

gmaurelia commented 6 years ago

Currently I have the same problem and nobody has been able to help me. In case it helps, I leave you these 2 links. The first is a demo of how to delegate from one host to another. And the second is the same problem you have.

Link1: https://asciinema.org/a/4nclzcuus3ubdcu88xmepz8u4 Link2: https://stackoverflow.com/questions/47887874/cant-signing-and-pushing-trust-metadata-in-notary

I hope it helps you,

Greetings and tell me how you are doing.

unullmass commented 6 years ago

@gmaurelia Glad to know that I'm not alone in this. I followed the first link you provided and the creator seems to have followed the exact same steps that I have. But the demo was created more than two years ago. So I presume older versions of Docker and Notary are being used in this case.

Have you been able to workaround this issue? Or is the only solution sharing of the root/targets keys?

ecordell commented 6 years ago

@unullmass In your docker push logs that you have several:

Enter passphrase for targets/release key with ID e55d5be:
Passphrase incorrect. Please retry.

It looks like the docker client is finding they but then is unable to sign because of of a failed passphrase. Just so we're clear - are these messages waiting for your input? if so are you using the same password you generated the key with?

unullmass commented 6 years ago

@ecordell Yes, I have been trying with the same passphrase which I had set a moment before. It is never accepted. Quite puzzling. I've even tried setting the passphrase to export NOTARY_DELEGATION_PASSPHRASE=mypasswordhere but that seems to be ignored by the Docker client.

gmaurelia commented 6 years ago

@ecordell I am automating this whole push process of an image and signature through the environment variables of both content trust and notary.

@unullmass in link 2 I suggest doing the delegation step with the current version of notary. However, I do what they tell me but now another error appears.

unullmass commented 6 years ago

@gmaurelia Which version of Docker client and Notary client/server are you working with?

I've been trying everything out with the freshest notary code off the Github master branch. The docker client and server are the ones available for Ubuntu 16.04.

ecordell commented 6 years ago

export NOTARY_DELEGATION_PASSPHRASE=mypasswordhere

I believe when using the docker client and not the notary client, the env var prefix changes to DOCKER_CONTENT_TRUST, so it would be DOCKER_CONTENT_TRUST_DELEGATION_PASSPHRASE.

Another suggestion: try running all commands with -D for debug output, we might see something there that helps.

gmaurelia commented 6 years ago

@unullmass this is the error that appears now after the solution of link 2. image

this my notary version: image

this my docker version:

image

a query. How do you have notary configured? that is, where they have located the notary client configuration file, the keys, etc. ??

@ecordell How do I see the logs of the docker push?

ecordell commented 6 years ago

"no valid signing keys for delegation role" sounds like either notary key import wasn't run or failed for some reason, or docker is not looking in the right directory for the keys. By default this is ~/.docker/trust (keys will be in private in that directory). You can double check that the key it's looking for is in that directory.

You can see the logs of a docker push by running with -D, e.g. docker -D push.

gmaurelia commented 6 years ago

@ecordell indeed, when consulting the private signature keys, this key can not be found. However, if I import it from the other host I get the same error:

image

unullmass commented 6 years ago

@ecordell Tried all the workarounds. The private delegation key is saved to the trust path and it is getting encrypted. But I think it is the decryption step which is failing and so I'm left with a key that is unusable. I've turned on the debug mode as suggested for Docker and Notary clients.

docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:24:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:23:00 2017
 OS/Arch:      linux/amd64
 Experimental: false
# notary version
notary
 Version:    0.5.0
 Git commit: a41821f

:~/notary_tmp# docker tag localhost:5000/mu_alpine:v1 localhost:5000/max_alpine:signed
:~/notary_tmp# notary init localhost:5000/max_alpine
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /root/.docker/trust 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /root/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] generated ECDSA key with keyID: 52a33d02e5fe08cd920a607d47b5acad2e8109db33fef7efaa08f16fa90f3b8e 
DEBU[0000] generated new ecdsa key for role: root and keyID: 52a33d02e5fe08cd920a607d47b5acad2e8109db33fef7efaa08f16fa90f3b8e 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID 52a33d0: 
Repeat passphrase for new root key with ID 52a33d0: 
DEBU[0003] No yubikey found, using alternative key storage: no library found 
DEBU[0003] generated ECDSA key with keyID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0003] generated new ecdsa key for role: targets and keyID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
Enter passphrase for new targets key with ID 35f9624: 
Repeat passphrase for new targets key with ID 35f9624: 
DEBU[0007] generated ECDSA key with keyID: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0007] generated new ecdsa key for role: snapshot and keyID: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
Enter passphrase for new snapshot key with ID 838523d: 
Repeat passphrase for new snapshot key with ID 838523d: 
DEBU[0016] got remote timestamp ecdsa key with keyID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0016] generating new snapshot...                   
DEBU[0016] Saving changes to Trusted Collection.        
DEBU[0016] signing root...                              
DEBU[0016] sign called with 1/1 required keys           
DEBU[0016] No yubikey found, using alternative key storage: no library found 
DEBU[0016] sign called with 0/0 required keys           
DEBU[0016] sign targets called for role targets         
DEBU[0016] sign called with 1/1 required keys           
DEBU[0016] No yubikey found, using alternative key storage: no library found 
DEBU[0016] sign called with 0/0 required keys           
DEBU[0016] signing snapshot...                          
DEBU[0016] sign called with 1/1 required keys           
DEBU[0016] No yubikey found, using alternative key storage: no library found 
DEBU[0016] sign called with 0/0 required keys

:~/notary_tmp# notary publish localhost:5000/max_alpine
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /root/.docker/trust 
Pushing changes to localhost:5000/max_alpine
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /root/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] received HTTP status 404 when requesting root. 
DEBU[0000] Loading trusted collection.                  
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] snapshot role has key IDs: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] verifying signature for key ID: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] applied 0 change(s)                          
DEBU[0000] sign targets called for role targets         
DEBU[0000] sign called with 1/1 required keys           
DEBU[0000] No yubikey found, using alternative key storage: no library found 
Enter passphrase for targets key with ID 35f9624: 
DEBU[0002] sign called with 0/0 required keys           
DEBU[0002] signing snapshot...                          
DEBU[0002] sign called with 1/1 required keys           
DEBU[0002] No yubikey found, using alternative key storage: no library found 
Enter passphrase for snapshot key with ID 838523d: 
DEBU[0005] sign called with 0/0 required keys           
Successfully published changes for repository localhost:5000/max_alpine

Now I'm going to add the delegation role:

:~/notary_tmp# notary delegation list locahost:5000/max_alpine
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /root/.docker/trust 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /root/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] cached snapshot is invalid (must download): sha256 checksum for snapshot did not match: expected 0b0a47432cb5f586bdca4a56141170bdd73140c17583f779cd3d235fd3c9ad2b 
DEBU[0000] 200 when retrieving metadata for snapshot.0b0a47432cb5f586bdca4a56141170bdd73140c17583f779cd3d235fd3c9ad2b 
DEBU[0000] snapshot role has key IDs: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] verifying signature for key ID: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] snapshot role has key IDs: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] verifying signature for key ID: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] successfully verified downloaded snapshot.0b0a47432cb5f586bdca4a56141170bdd73140c17583f779cd3d235fd3c9ad2b 
DEBU[0000] Loading targets...                           
DEBU[0000] cached targets is invalid (must download): sha256 checksum for targets did not match: expected 3c1f62a187dcfee7699f5c94f4e958eeaeabebc06cba630fe49d6f9a86de1e21 
DEBU[0000] 200 when retrieving metadata for targets.3c1f62a187dcfee7699f5c94f4e958eeaeabebc06cba630fe49d6f9a86de1e21 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] successfully verified downloaded targets.3c1f62a187dcfee7699f5c94f4e958eeaeabebc06cba630fe49d6f9a86de1e21 

No delegations present in this repository.

Rotating the snapshot key just in case:

:~/notary_tmp# notary key rotate localhost:5000/max_alpine snapshot -r 
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /root/.docker/trust 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /root/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] 200 when retrieving metadata for root        
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] cached snapshot is invalid (must download): sha512 checksum for snapshot did not match: expected 922bbde1576a0a9a6f0d69a57a932d96d78d7a75620029bc0ec2f751f396c6ad358ce910101c0347c250d63ed364292c3cade0026459276430fdbc46ff687bad 
DEBU[0000] 200 when retrieving metadata for snapshot.fd41650ad77237a5f90368b189081d93513be4418dee8b6be761dde0394d80e7 
DEBU[0000] snapshot role has key IDs: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] verifying signature for key ID: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] snapshot role has key IDs: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] verifying signature for key ID: 838523d1f1d16f30d664706b7faaebbb77ac3075d1693a5033ae6faeb1664fd1 
DEBU[0000] successfully verified downloaded snapshot.fd41650ad77237a5f90368b189081d93513be4418dee8b6be761dde0394d80e7 
DEBU[0000] Loading targets...                           
DEBU[0000] cached targets is invalid (must download): sha256 checksum for targets did not match: expected 2cc7b9b9283dd8ecca0b5ca09fbaec589c93e6af130f9fdcef99bf46cd8d5673 
DEBU[0000] 200 when retrieving metadata for targets.2cc7b9b9283dd8ecca0b5ca09fbaec589c93e6af130f9fdcef99bf46cd8d5673 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] successfully verified downloaded targets.2cc7b9b9283dd8ecca0b5ca09fbaec589c93e6af130f9fdcef99bf46cd8d5673 
DEBU[0000] skipping targets/releases because there is no checksum for it 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] applied 1 change(s)                          
DEBU[0000] signing root...                              
DEBU[0000] sign called with 1/1 required keys           
DEBU[0000] No yubikey found, using alternative key storage: no library found 
Enter passphrase for root key with ID 52a33d0: 
DEBU[0002] sign called with 0/0 required keys           
DEBU[0002] signing snapshot...                          
DEBU[0002] sign called with 1/1 required keys           
DEBU[0002] No yubikey found, using alternative key storage: no library found 
DEBU[0002] Client does not have the key to sign snapshot. Assuming that server should sign the snapshot. 
Successfully rotated snapshot key for repository localhost:5000/max_alpine

Adding the delegation role using user x509 cert:

# notary delegation add localhost:5000/max_alpine targets/releases collabo.crt --all-paths -p
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /root/.docker/trust 
WARN[0000] certificate with CN  is near expiry          
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /root/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] Adding delegation "targets/releases" with threshold 1, and 1 keys
DEBU[0000] Adding [] paths to delegation targets/releases\n 

Addition of delegation role targets/releases with keys [35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0], with paths ["" <all paths>], to repository "localhost:5000/max_alpine" staged for next publish.

DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /root/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
Auto-publishing changes to localhost:5000/max_alpine
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] 200 when retrieving metadata for root        
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] no snapshot in cache, must download          
DEBU[0000] 200 when retrieving metadata for snapshot.dfa39b823ed8aeed59dffaaae7f797b515ff4d247165d218b3b3de23b2c8b7bb 
DEBU[0000] snapshot role has key IDs: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] verifying signature for key ID: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] successfully verified downloaded snapshot.dfa39b823ed8aeed59dffaaae7f797b515ff4d247165d218b3b3de23b2c8b7bb 
DEBU[0000] Loading targets...                           
DEBU[0000] no targets in cache, must download           
DEBU[0000] 200 when retrieving metadata for targets.a46aff41bdcf0410e3950997861e8fb6eb007af1b052b812aa1425cd5b9e3e9b 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] successfully verified downloaded targets.a46aff41bdcf0410e3950997861e8fb6eb007af1b052b812aa1425cd5b9e3e9b 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
Enter passphrase for targets key with ID 35f9624: 
DEBU[0002] role targets/releases with no Paths will never be able to publish content until one or more are added 
DEBU[0002] No yubikey found, using alternative key storage: no library found 
DEBU[0002] No yubikey found, using alternative key storage: no library found 
DEBU[0002] No yubikey found, using alternative key storage: no library found 
DEBU[0002] applied 2 change(s)                          
DEBU[0002] sign targets called for role targets         
DEBU[0002] sign called with 1/1 required keys           
DEBU[0002] No yubikey found, using alternative key storage: no library found 
DEBU[0002] sign called with 0/0 required keys           
DEBU[0002] signing snapshot...                          
DEBU[0002] sign called with 1/1 required keys           
DEBU[0002] No yubikey found, using alternative key storage: no library found 
DEBU[0002] Client does not have the key to sign snapshot. Assuming that server should sign the snapshot. 
Successfully published changes for repository localhost:5000/max_alpine

We verify that the delegation role has been added:

:~/notary_tmp# notary delegation list localhost:5000/max_alpine
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /root/.docker/trust 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] Making dir path: /root/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config for localhost:5000/max_alpine 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] cached snapshot is invalid (must download): sha512 checksum for snapshot did not match: expected e4757dc062841cc3e8607f69326709144323e7e4630a249789c33eae482f816b5616bc893b57e327bfb734c20dfef877804194bb2d3fe706d819e9c1199b7153 
DEBU[0000] 200 when retrieving metadata for snapshot.652df6fae2a1e65e097de62adc626dc7edb03b6a2b9f2eb95255207e6edf5777 
DEBU[0000] snapshot role has key IDs: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] verifying signature for key ID: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] snapshot role has key IDs: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] verifying signature for key ID: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] successfully verified downloaded snapshot.652df6fae2a1e65e097de62adc626dc7edb03b6a2b9f2eb95255207e6edf5777 
DEBU[0000] Loading targets...                           
DEBU[0000] cached targets is invalid (must download): sha256 checksum for targets did not match: expected 9d0ec0102ce73c5f4c6aedf97299b0fbfe9e0051611b5495bf03e109d55427a1 
DEBU[0000] 200 when retrieving metadata for targets.9d0ec0102ce73c5f4c6aedf97299b0fbfe9e0051611b5495bf03e109d55427a1 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] successfully verified downloaded targets.9d0ec0102ce73c5f4c6aedf97299b0fbfe9e0051611b5495bf03e109d55427a1 
DEBU[0000] skipping targets/releases because there is no checksum for it 

ROLE                PATHS             KEY IDS                                                             THRESHOLD
----                -----             -------                                                             ---------
targets/releases    "" <all paths>    35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0    1

Now on the collaborators machine, we import the private key:

$ notary key import collabo.key --role targets/releases
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /home/unullmass/.docker/trust 
Enter passphrase for new targets/releases key with ID 35c0fef: 
Repeat passphrase for new targets/releases key with ID 35c0fef: 

$ notary key list
DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /home/unullmass/.docker/trust 
DEBU[0000] No yubikey found, using alternative key storage: no library found 

DEBU[0000] No yubikey found, using alternative key storage: no library found 
ROLE                GUN    KEY ID                                                              LOCATION
----                ---    ------                                                              --------
targets/releases           35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0    /home/unullmass/.docker/trust/private

$ ls /home/unullmass/.docker/trust/private/35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0.key 
/home/unullmass/.docker/trust/private/35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0.key

We see that signing key has the 35c0fef0 id. Now for the final push-and-sign:

$ export DOCKER_CONTENT_TRUST_DELEGATION_PASSPHRASE=password123
$ docker -D push localhost:5000/max_alpine:signed
The push refers to a repository [localhost:5000/max_alpine]
37fad0b98dd5: Layer already exists 
074037175d22: Layer already exists 
04a094fe844e: Layer already exists 
signed: digest: sha256:14e70551175642b857b13465a695d7a9d1aa7cbbb5a06cd556d451bc4e6a7ba8 size: 951
Signing and pushing trust metadata
DEBU[0000] reading certificate directory: /home/unullmass/.docker/tls/localhost:4443 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config%!(EXTRA string=localhost:5000/max_alpine) 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config%!(EXTRA string=localhost:5000/max_alpine) 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] snapshot role has key IDs: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] verifying signature for key ID: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] successfully verified cached snapshot        
DEBU[0000] Loading targets...                           
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] successfully verified cached targets         
DEBU[0000] skipping targets/releases because there is no checksum for it 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config%!(EXTRA string=localhost:5000/max_alpine) 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config%!(EXTRA string=localhost:5000/max_alpine) 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] snapshot role has key IDs: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] verifying signature for key ID: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] successfully verified cached snapshot        
DEBU[0000] Loading targets...                           
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] successfully verified cached targets         
DEBU[0000] skipping targets/releases because there is no checksum for it 
DEBU[0000] Making dir path: /home/unullmass/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] Adding target "signed" with sha256 "14e70551175642b857b13465a695d7a9d1aa7cbbb5a06cd556d451bc4e6a7ba8" and size 951 bytes.

DEBU[0000] Making dir path: /home/unullmass/.docker/trust/tuf/localhost:5000/max_alpine/changelist 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config%!(EXTRA string=localhost:5000/max_alpine) 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] entered ValidateRoot with dns: localhost:5000/max_alpine 
DEBU[0000] found the following root keys: [f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175] 
DEBU[0000] found 1 valid leaf certificates for localhost:5000/max_alpine: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] found 1 leaf certs, of which 1 are valid leaf certs for localhost:5000/max_alpine 
DEBU[0000] checking root against trust_pinning config%!(EXTRA string=localhost:5000/max_alpine) 
DEBU[0000] checking trust-pinning for cert: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000]  role has key IDs: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] verifying signature for key ID: f42f16d844b515c68eef8d06419730373dfc69e4f12ad63df234e0f780cb1175 
DEBU[0000] root validation succeeded for localhost:5000/max_alpine 
DEBU[0000] 200 when retrieving metadata for root        
DEBU[0000] updating TUF client                          
DEBU[0000] Loading timestamp...                         
DEBU[0000] 200 when retrieving metadata for timestamp   
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] timestamp role has key IDs: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] verifying signature for key ID: 1304fa0560db6ab9494f1b3af599c256c585130cd0c477da942f4c8cec022d0f 
DEBU[0000] successfully verified downloaded timestamp   
DEBU[0000] Loading snapshot...                          
DEBU[0000] snapshot role has key IDs: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] verifying signature for key ID: 911fa636e15c2eaa3f62f42091e55a75ba3cef3ad08e90140b7630357e1398a9 
DEBU[0000] successfully verified cached snapshot        
DEBU[0000] Loading targets...                           
DEBU[0000] targets role has key IDs: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] verifying signature for key ID: 35f9624839a8aa08fd74ae7e58efea768447c6a4b380338628f93075d92b277b 
DEBU[0000] successfully verified cached targets         
DEBU[0000] skipping targets/releases because there is no checksum for it 
DEBU[0000] changelist add: signed                       
WARN[0000] certificate with CN  is near expiry          
DEBU[0000] No yubikey found, using alternative key storage: no library found 
DEBU[0000] No yubikey found, using alternative key storage: no library found 
Enter passphrase for targets/releases key with ID 35c0fef: 
Passphrase incorrect. Please retry.
Enter passphrase for targets/releases key with ID 35c0fef: 
Passphrase incorrect. Please retry.
Enter passphrase for targets/releases key with ID 35c0fef: 
Passphrase incorrect. Please retry.
Enter passphrase for targets/releases key with ID 35c0fef: 
ERRO[0029] couldn't add target to targets/releases: could not find necessary signing keys, at least one of these keys must be available: 35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0 
DEBU[0029] error attempting to apply change #0: create, on scope: targets/releases path: signed type: target 
DEBU[0029] Error applying changelist                    
Failed to sign "localhost:5000/max_alpine":signed - could not find necessary signing keys, at least one of these keys must be available: 35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0
Error: could not find signing keys for remote repository localhost:5000/max_alpine, or could not decrypt signing key: could not find necessary signing keys, at least one of these keys must be available: 35c0fef0178118e314542e1b83937655caeb85397f9857d31bcf4aca1d881de0

At this stage all I can think of is something preventing the signing private key from being decrypted.

gmaurelia commented 6 years ago

The problem I had was that before I docker push, I applied the command: notary init my.registry:443/collection so notary generated a collection with different keys and in this way I could not do push docker of any image under any role nor even targets.

Once I did it the right way, I applied the steps you mentioned to me and the problem was solved. The notary configuration is the following:

command: tree $HOME/.docker/trust/

.docker/trust
├── certs
│   ├── delegation.crt
│   └── proof
│       ├── delegation.crt
│       ├── delegation.csr
│       └── delegation.key
├── config.json
├── private
│   ├── root_keys
│   │   └── 4e46a197de40621094f86e0cea4aa892d7c3cfb1b3400c64f6d7d82e4b97a470.key
│   └── tuf_keys
│       ├── 3269a0858ca91001c543435d0242e747bd08e68b52533f1b42028388ed02c7e6.key
│       └── my.registry:443
│           └── galera-leader-proxy
│               └── 
|           873ba8267df2be149fba2230441961812159c35537b18c133247239f4bafa989.key
├── root-ca.crt
├── tls
│   └── my.registry:443
│       └── root-ca.crt
└── tuf
    └── my.registry:443
        └── galera-leader-proxy
            ├── changelist
            └── metadata
                ├── root.json
                ├── snapshot.json
                ├── targets
                │   ├── kube1.json
                │   └── releases.json
                ├── targets.json
                └── timestamp.json

On the other hand, to configure the client correctly I defined the following alias:

alias dockernotary="notary -c $HOME/.docker/trust/config.json -d $HOME/.docker/trust/ -s https://notary-server:4443"

Saludos.

liamawhite commented 6 years ago

I am also seeing similar things when trying to rotate keys. I get passphrase incorrect on my root key even though it's definitely correct.

notary
 Version:    0.4.3
 Git commit: 9211198
gmaurelia commented 6 years ago

@liamawhite I would recommend that you do not rotate the keys, except in case of contingency or have a security problem.

On the other hand, you checked that the keys are the same ?. I recommend that you use the tree command (from the post above) to see the keys you have on your host.

Personally I decided not to use a rotate for the same reason. Notary is a very unstable service that is still in development.

Greetings and I remain attentive.

liamawhite commented 6 years ago

@gmaurelia so I've been doing some more playing around and it seems to work before I run a notary delete, but when I try to rotate a key after a delete it then says that my pass phrase is incorrect.

The same behaviour occurs if I do a delete by rm -rf ~/.docker/trust and delete everything from my database.

Edit: this behaviour isn't limited to just key rotation either. It's everything that involves put/posting to the server.

Double edit: this also isn't limited to our server and signer. We have recreated it with Docker Hub.

gmaurelia commented 6 years ago

@liamawhite Remember that to delete a notary content you must do it locally and remotely. For this, the command to use is: notary delete <collection> --remote

when I want to delete all the contents of a collection I execute these two commands:

notary delete my.registry:443/image
notary delete my.registry:443/image --remote
liamawhite commented 6 years ago

We have done this both manually and with that command

cyli commented 6 years ago

Are you all using the 0.5.0 binary downloaded from the releases page, or building notary master? We had updated the encryption format for the keys, but that change never made it into any release of notary. It did however make it into a vendor into docker.

So docker >= 17.12, notary master, and now notary 0.6.0 will generate keys using the new format, which looks like:

-----BEGIN ENCRYPTED PRIVATE KEY-----
MIHOMEkGCSqGSIb3DQEFDTA8MBsGCSqGSIb3DQEFDDAOBAjsfXVXuwOQnAICCAAw
HQYJYIZIAWUDBAEqBBB0O793rOzupOUavjLSiPmBBIGALJxsXCe8rLBfeviStIH0
A+1jCXUqXNm8D4npyNui/JRi/CjYPqgcO/2ulP8ppUAeTnLVQdhpv5ZOemK5ibMc
ECaNuzo40snnpve4duZEufkI9hXrO6MAMRT+G5ep1rKyIKboIPkzYUAdezj5ggUu
p1Gc8HB7j2SYjQX0Ybvlr6k=
-----END ENCRYPTED PRIVATE KEY-----

This format is not readable by notary < 0.6.0 and docker < 17.12, which generates keys which look like:

-----BEGIN EC PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,3e655dfb331a6d2c735edaa33d55d09a
role: root

UFAyzUyzA4htkgcvb+ZYa5Y7drX+OaJXHbZgYqawxH/OdY4mQ2bpLfExYCUJciL3
dmO5Qafrc6syYW0sk4pb/vdr/dbjmLK9IJjxZIHuv7NRqvLjnRq0C8RX+FYv3lGm
RgLLfnEKDm/gQvHWmb0rseBaWn3Ww7oXPnU5qQN/Cv4=
-----END EC PRIVATE KEY-----

Docker >= 17.12, notary master, and notary 0.6.0 can read the keys in this format, however, so they are backwards compatible with previous versions, just not forwards compatible.

Could that be the issue? If so, many apologies for the confusion - I've tried to document it here. https://github.com/theupdateframework/notary/pull/1311.

ankitsrao commented 3 years ago

Hi Team,

I'm maybe facing a simliar kind of issue.

Below are the steps I performed to sigh my image:

a) Generated signing keys, using below command: docker trust key generate demo-key

b) Allowed the public key to sign images docker trust signer add --key demo-key.pub demo-key <docker-repository/image>

c) Sign the image, using: docker trust sign <docker-repository/image:tag> When doing so, I receive the below error: image

Note : I can see the signers and keys added using the docker trust inspect command.

Please do let me know, if I'm missing anything.

Thanks.

Monishguru commented 3 years ago

@ankitsrao Can you please let me know how you fixed this?