Support multiple notary servers and/or fallback logic

[ruimarinho]

It would be useful to support multiple notary servers (ex: official Docker hub server + private notary server) without having to prefix each command with DOCKER_CONTENT_TRUST_SERVER to docker or different -s flags to notary.

One solution could be to map the notary servers per repositories in the client config. Another possible (albeit partial) solution to this could be intelligent fallback to a different notary server based on the registry name.

Original issue: When building from a CI system, I need to have DOCKER_CONTENT_TRUST_SERVER set to my private notary server, but that means building private images based on signed tags requires switching back and forth notary servers (otherwise I get trusted data does not exist when using FROM alpine).

[endophage]

Would it work for you to prefix the command with the server? For example

$> DOCKER_CONTENT_TRUST_SERVER=https://notary.docker.io docker build ..."

That will temporarily override your server setting for the one command.

[ruimarinho]

Not really, unless I'm missing something. Consider the following Dockerfile of my app:

FROM alpine
COPY . ./

In order to build this on the CI system, I would need to have trust data for the docker.io/library alpine image, whose metadata is served by notary.docker.io. However, the build would have to be signed using a custom notary server. Unless I run two passes of docker build (pointing to two different DOCKER_CONTENT_TRUST_SERVER), then I don't think it solves this issue.

[cyli]

@ruimarinho If your Dockerfile points to alpine, I think you don't need the DOCKER_CONTENT_TRUST_SERVER to point to your private notary server, because as you say, you want to get the trust data from notary.docker.io.

However, I'm not sure I understand why you would need a second pass of docker build that does point to your own private notary server? Could you elaborate on this use case? Do you mean that you are building 2 images, and one is based on the official alpine, and the second is based on a newly-built image that you signed?

[ruimarinho]

@cyli, I was under the impression that docker build would need to have access to the private notary server to build the signed image locally (on the CI system), but from my tests that does not seem to be the case. So DOCKER_CONTENT_TRUST_SERVER=https://notary.docker.io docker build ..., as suggested, will work as it pulls any publicly signed image from Docker Hub, and then I can let docker push use the exported DOCKER_CONTENT_TRUST_SERVER environment variable pointing to the private notary server.

Still, not having to muck around with these would be simpler :) also not sure of the security implications of having a private notary server proxying requests to an "upstream" server.

[cyli]

@ruimarinho Ah right sorry, docker build does not actually do any signing - it just builds a local image that currently doesn't really have any trust implications. You can't run that image with content trust enabled, for instance.

Only docker push does the signing, because the final image digest is only created when pushing (since the digest is of the manifest, which contains the digests of the compressed layers, and the compression only happens when you push), and that image digest is what gets signed into the trust data.

[ruimarinho]

Thanks for the clarification @cyli. So I think the only thing keeping this issue open is wether it would still make sense to proxy requests or not.

[cyli]

@ruimarinho Please correct me if I'm misinterpreting, but it seems that you are proposing proxying as a solution to the more general issue of being able to support multiple notary servers, possibly intelligent fallback based on the registry name?

If so, it might be good to discuss that feature instead, and the details of how it should work and what the expected behaviors are, in this issue instead, in case a different solution would make more sense (for example, a client config mapping a list of notary servers to try to pull from). I think we discussed it at some point, but no official issue ever got filed, so this could be the place for it. :) Alternately we can open a different, more general issue.

[ruimarinho]

That's exactly it :). Feel free to rename the title to whatever you think would be a more appropriate and/or recognisable by the notary team.

[riyazdf]

@ruimarinho: thanks, updated! Please feel free to add/edit as necessary :)

[zparnold]

Has this been addressed at all? Would this change need to happen on the notary client, in the docker client or in the docker runtime? Happy to assist if necessary. @riyazdf

[cyli]

This has not been addressed, @zparnold. Docker content trust currently happens in the docker client, not the engine runtime. To be useful for docker pull or docker push it'd have to be on the docker CLI - I don't think any changes need to be made to the notary client in the mapping case, because you are just deciding which server to use based on the GUN (e.g. docker.io/library/alpine), but possibly for fallback behavior (where you try one notary server, and if there are any errors you fall back to another, etc.) some better error reporting or transactional cache writing could be added to the notary client library as well.

[zparnold]

@cyli Cool, I'll head over there first! :) Thanks!