Need mechanism for registering certs for private docker registries

rcernich commented 9 years ago

Currently, using a private docker registry in an image stream produces the following output in the openshift master instance logs:

22:41:12 ce-openshift-master openshift[2579]: E0414 22:41:12.285476    2579 factory.go:51] error getting X-Docker-Token from index.docker.io: Get https://private-registry/v1/repositories/namespace/name/images: x509: certificate signed by unknown authority

In this case, there is a cert in /etc/docker/certs.d/private-registry/. If that could be used, that would be great, but there should be some way to register the cert with openshift.

ncdc commented 9 years ago

Note, this is for registries that use certificates.

If we do this, we'll need to have a way for users/admins to supply certs, since /etc/docker/certs.d is host specific.

smarterclayton commented 9 years ago

This is being worked on now - service accounts will provide pull secrets, and then image streams that point to remote docker repos will allow you to specify which service account should be used for that repo.

liggitt commented 9 years ago

pull secrets won't get you per-pod certs, only dockercfg-level credential (user/password). custom certs (or ca?) for registries would probably still be node-wide. @deads2k fyi

ncdc commented 9 years ago

Registries can run with a cert signed by a nonstandard/nonroot CA. In this case, anything that talks to the registry needs to know about the CA that signed the registry's cert. There are 2 components that I'm aware of that need to know about the CA:

The Docker daemon, which contacts the registry for pushes and pulls
OpenShift's ImageController, which imports tag and image information from external registries

Docker will look in /etc/docker/certs.d/<registry URL>/ for any file ending with .crt and add that to the cert pool it uses for verification. If the registry directory doesn't have any .crt files, Docker will fall back to using the system root CA bundle from /etc/pki/tls/.... Either way, in order for Docker to use a custom CA cert, the node's sysadmin must install the custom CA file; an end-user can't pass a custom CA when creating an image stream.

We can either modify OpenShift to use the same logic as Docker, or we could recommend that custom CA certs be installed in the root CA bundle, in which case they'd be available for both OpenShift and Docker to use.

smarterclayton commented 9 years ago

The CA for OpenShift should be considered trusted by the nodes in the cluster, so the CA should be added to all the nodes and masters.

----- Original Message -----

Registries can run with a cert signed by a nonstandard/nonroot CA. In this case, anything that talks to the registry needs to know about the CA that signed the registry's cert. There are 2 components that I'm aware of that need to know about the CA:

The Docker daemon, which contacts the registry for pushes and pulls

OpenShift's ImageController, which imports tag and image information from external registries

Docker will look in /etc/docker/certs.d/<registry URL>/ for any file ending with .crt and add that to the cert pool it uses for verification. If the registry directory doesn't have any .crt files, Docker will fall back to using the system root CA bundle from /etc/pki/tls/.... Either way, in order for Docker to use a custom CA cert, the node's sysadmin must install the custom CA file; an end-user can't pass a custom CA when creating an image stream.

We can either modify OpenShift to use the same logic as Docker, or we could recommend that custom CA certs be installed in the root CA bundle, in which case they'd be available for both OpenShift and Docker to use.

Reply to this email directly or view it on GitHub: https://github.com/openshift/origin/issues/1753#issuecomment-100231345

ncdc commented 9 years ago

Yes.... but what about random 3rd party registries that have a cert signed by some other CA?

smarterclayton commented 9 years ago

Add them to all the nodes and masters as a trusted root CA.

----- Original Message -----

Yes.... but what about random 3rd party registries that have a cert signed by some other CA?

Reply to this email directly or view it on GitHub: https://github.com/openshift/origin/issues/1753#issuecomment-100245851

smarterclayton commented 8 years ago

With pull through we could support this as an additional secret type. That and import would then work, removing the need to push the carts to the node.

liggitt commented 8 years ago

Can you isolate the certs used per pull? I thought that was the issue, that one pull's trusted CA certs affected others.

smarterclayton commented 8 years ago

We can select a transport per destination - but you're right, we'd have to know the destination from the cert efficiently.

On Jan 30, 2016, at 8:28 AM, Jordan Liggitt notifications@github.com wrote:

Can you isolate the certs used per pull? I thought that was the issue, that one pull's trusted CA certs affected others.

— Reply to this email directly or view it on GitHub https://github.com/openshift/origin/issues/1753#issuecomment-177177188.

smarterclayton commented 8 years ago

I'm debating whether I want this to be a cluster wide setting for import (everyone can import from all clusters with these CAs). At a minimum, it seems like admins want to be able to provide the cluster wide CA to import (which I'm doing) but multiple client keys per request isn't possible using our normal client today so we'd have to do something slightly different there.

martinpitt commented 7 years ago

I've been fighting with this for half a day now. I'm trying to convince oc import-image to pull a container from a local docker registry instead of from hub.docker.com. Note that I can not use --insecure=true, I'm not really testing the oc command but cockpit's Openshift integration. But the openshift/origin-docker-registry container:

doesn't read /etc/sysconfig/docker from either the host or the container (for specifying INSECURE_REGISTRY)
doesn't read /etc/docker/certs.d/ from either the host or the container (for specifying the certificate)
doesn't accept docker container registries whose SSL certificate is signed by OpenShift's CA (it would really be nice to do that by default)
/dockerregistry does not accept any command line argument like --insecure-registry so that it could be overridden with an --entrypoint
/dockerregistry does not seem to have an obvious way of reloading its config files (`/etc/pki/*), at least not with SIGHUP -- it just crashes
stopping the container and starting one manually with first copying the cert into the container's /etc/pki/ or bind-mounting that from the host doesn't work either, as the openshift controller immediately restarts its own instance

Did I miss anything? Is there any workaround or hack that I missed? Thanks in advance for any hint!

martinpitt commented 7 years ago

In case someone else stumbles over this: The only way how to make this work properly seems to be to rebuild the Openshift registry container with the CA already included in /etc/pki/:

mkdir /tmp/registry
cd /tmp/registry
cat << EOF > Dockerfile
FROM openshift/origin-docker-registry:$VERSION
ADD *.crt /etc/pki/ca-trust/source/anchors/
USER 0
RUN update-ca-trust extract
USER 1001
EOF
cp /openshift.local.config/master/ca.crt openshift-ca.crt
docker build --tag openshift/origin-docker-registry:$VERSION .
cd /tmp/
rm -r /tmp/registry

$VERSION is something like v1.5.0.

schweikert commented 7 years ago

I am still struggling with this... Can you help me explain how this is supposed to work? When executing "oc new-app", an API call to the openshift master is made, right?

With loglevel=8 I see something like this:

 dockerimagelookup.go:220] Docker registry lookup failed: Get https://docker-registry.....:5000/v2/: x509: certificate signed by unknown authority

Who is doing the SSL verification? Isn't it the openshift process running on master? Or is it in reality the registry doing the SSL verification? And if that is the case, what is the API call to the openshift registry made?

In other words: do I need to fix the SSL trust in the openshift3/ose container or in openshift3/ose-docker-registry?

miminar commented 7 years ago

@schweikert I presume the https://docker-registry.....:5000/v2/ refers to your integrated registry and it's secured.

The oc new-up command itself attempts to query the registry. Is your docker-registry signed with the same CA as your OpenShift cluster? If not, it may be the cause of this problem.

We should definitely allow you to specify docker-registry CA if secured and different from the cluster one. AFAIK we don't. You might be able to overcome it using --certificate-authority='/path/to registry/ca' --insecure-skip-tls-verify=true which will render at least one connection insecure (either to the master API or to the registry).

schweikert commented 7 years ago

Thanks for the hints! I can overcome the problem in 'oc new-app' by using '--insecure-registry=true', but the OpenShift UI for importing images doesn't have that option. Also, it seem to me that it isn't the "oc" command-line client that does the verification, but it is either the master process doing it, or possibly doing it via the internal registry.

schweikert commented 7 years ago

After endless trying, I found a workaround to my problem:

edit /etc/systemd/system/atomic-openshift-master.service, and add to ExecStart:

-v /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted
reload systemctl, restart the service

Now /etc/pki/ca-trust/extracted from the docker host is mapped inside the ose container, and SSL verification works.

miminar commented 7 years ago

@schweikert Good point, there's --insecure-registry=true as well. I missed that one.

Also you're right about the master server generating the error. oc new-app delegates the import attempt it to the the master API. Since you're running the services in containers, I don't think I can provide a better solution.

nelsong commented 7 years ago

@schweikert Thanks a million, this worked like a charm, been struggling with that for a few days.

bparees commented 7 years ago

@miminar whether or not we get to the point of adding a mechanism for this, we desperately need official documentation for how a user can add trust for a registry with a self-signed cert, covering all the places they need to update and where to put the certificates on each:

1) all the docker daemons on all the hosts 2) the master server host 3) the docker registry pod

bparees commented 7 years ago

This is at least a p2 to get decent documentation in place, we need better guidance for users trying to access third party registries.

jefflee416 commented 6 years ago

@schweikert Thanks for sharing. I tried to add the option to ExecStart, but openshift complaining "-v" isn't a valid "openshift start " option.

bparees commented 6 years ago

-v is an arg to docker, not open shift. It's for mounting volumes into a container.

jefflee416 commented 6 years ago

Correct, "-v" is an arg to "docker run" to map host volume to containers, but it is not a valid arg to ExecStart of /etc/systemd/system/multi-user.target.wants/docker.service either. So I wonder how they make it work.

schweikert commented 6 years ago

Note that this was recently fixed and works for me without hacking with OpenShift Origin 3.6. See: https://github.com/openshift/openshift-ansible/pull/4414

jefflee416 commented 6 years ago

Thanks David. I figured out why I don't see this option in my systemd files, I installed the cluster as 'native' cluster, which led to role "/usr/share/ansible/openshift-ansible/roles/openshift_master/templates/native-cluster" being triggered.

miminar commented 6 years ago

The openshift/openshift-ansible#4414 PR won't make pullthrough work unless istag or imagestream is marked as insecure. We still need to provide documentation for propagating the certificates into registry pod.

openshift-bot commented 6 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

bparees commented 6 years ago

/remove-lifecycle-stale /lifecycle frozen

as far as i know we still need better doc in this space.

bparees commented 6 years ago

/remove-lifecycle stale

ttindell2 commented 6 years ago

Our project has currently run into this issue also. It would be great if there was a simple way to add ca certs to the docker registry image

marcpopp commented 6 years ago

We also ran into this problem. I could not find any solution in this or linked tickets? How is this done?

ttindell2 commented 6 years ago

What we did as a workaround is mount ca-bundle as a secret to the registry deployment:

Create the secret. In our case, we just attach the ca bundle on the server currently running the oc commands. Make sure to add the ca to your server if you want them added to the registry ca:

oc secrets new --namespace=default registry-ca-bundle /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

Attach the secret to the docker-registry:

oc volume dc/docker-registry --add --namespace=default --type=secret --name=registry-ca-bundle --secret-name=registry-ca-bundle --mount-path=/etc/pki/ca-trust/extracted/pem

bparees commented 6 years ago

Our project has currently run into this issue also. It would be great if there was a simple way to add ca certs to the docker registry image

@ttindell2 @marcpopp this is a different problem and requires a different configuration solution, and it is being tracked here: https://github.com/openshift/openshift-docs/issues/8375

bparees commented 6 years ago

The issue of how to import images from a registry w/ an untrusted certficate was documented here, so i'm closing this: https://docs.openshift.org/latest/dev_guide/managing_images.html#trusting-registries

/close

samir-saad commented 6 years ago

According to docs: https://docs.openshift.com/container-platform/3.7/day_two_guide/docker_tasks.html#import-images-from-external-registries

1- Add the external registry certificate to /etc/pki/ca-trust/source/anchors/ $ sudo cp /etc/pki/ca-trust/source/anchors/

2- sudo update-ca-trust

3- sudo systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers

akostadinov commented 5 years ago

How does one do this for v4.x cluster?

bparees commented 5 years ago

@akostadinov https://github.com/openshift/origin/issues/1753#issuecomment-377929424

akostadinov commented 4 years ago

I later found doc about adding registries with custom CAs [1] and it works.

It will be useful to have ability to add CAs without linking to specific registry in the case one has multiple repos with same trusted CA that is not present in standard operating system CAs.

The document you list sounds like manually editing node file system. But in 4.x nodes are managed by OpenShift. Is it for 3.x? Or did I miss something?

[1] https://docs.openshift.com/container-platform/4.2/registry/configuring-registry-operator.html

bparees commented 4 years ago

The document you list sounds like manually editing node file system. But in 4.x nodes are managed by OpenShift. Is it for 3.x? Or did I miss something?

you're correct, those were 3.x docs, i should have double checked the link. For 4.x there's this (what i think you were linking to above) which as you note requires a specific entry for each registry: https://docs.openshift.com/container-platform/4.2/registry/configuring-registry-operator.html#registry-operator-config-resources-configmap_configuring-registry-operator

and also this for configuring additional CAs for the cluster as a whole..(it's specifically intended for use with a proxy, but the CAs you specify there are consumed by many components, including nodes, and will be used when talking to registries): https://docs.openshift.com/container-platform/4.2/networking/enable-cluster-wide-proxy.html#nw-proxy-configure-object_config-cluster-wide-proxy

even if you do not configure a proxy, you should be able to configure the trustedCAs portion of the proxy config.

openshift / origin

Need mechanism for registering certs for private docker registries #1753