Can’t push to registry with self signed certificate

[gregseth]

I’m using Singularity v3.9.0.
I have a Docker registry running with a self-signed certificate.
I managed to be able to pull images from this registry by adding the following to ~/.config/containers/registries.conf (domain.local being replaced by the actual domain):

[[registry]]
prefix = "*.domain.local"
insecure = true

But if I try to push an image to the registry I get the following error:

singularity push -U image.sif oras://registry.domain.local/image-sgl
FATAL: Unable to push image to oci registry: unable to push: failed to do request: Head "https://registry.domain.local/v2/image-sgl/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a": x509: certificate signed by unknown authority

[dtrudg]

Hi @gregseth - thanks for the report. This is actually a known issue that should have already been filed by ourselves. Unfortunately it's a bit more complex to address than it appears at first... but we'll put some effort in considering how we handle it in the short term and/or longer term...

The cause of this differential behaviour is that while docker:// pulls use containers/image functionality (which reads docker/config.json and containers/registries.conf), the oras:// code uses the oras-project/oras-go client, which does not use containers/image. Instead it is using containerd/remotes/docker where host specific options must be provided in an explicit manner.

We do have a way of providing auth for an oras push via singularity remote login docker:// and there's an --insecure flag on that - but it does not work with oras://, nor does it allow specifying an "insecure registry" without authentication.

Perhaps for the short term we could:

1. Ensure the --insecure flag to remote login honored properly for authenticated oras:// operations
2. Introduce an --insecure flag that sets insecure mode for a registry in oras operations and other pulls. This would allow insecure pulls without authenticatin.

An --insecure flag would have slightly different semantics than the existing --http-only flag, as an insecure registry in Docker parlance allows both https with invalid certificates, and http... which it tries in that order. It might make sense to actually deprecate --http-only for consistency with other tools, and internally (see remote add --insecure). Need to consider shub, library, net sources also.

Longer term I believe we broadly favour moving away from the oras-project/oras-go client due to the mismatch of its dependencies and behavior with other push/pull functionality in Singularity. Because of the recent ORAS work in Helm there is increased interest in containers/image being able to handle non OCI image artifacts. We may be able to leverage this in future: https://github.com/containers/image/issues/1370

Pinging @tri-adam for any thoughts.

Quick setup for investigation....

mkdir -p ssreg/certs
cd ssreg
openssl genrsa 4096 > certs/domain.key
openssl req -new -x509 -nodes -sha1 -days 365 -key certs/domain.key -out certs/domain.crt
docker run -d   --name registry   -v `pwd`/certs:/certs   -e REGISTRY_HTTP_ADDR=0.0.0.0:443   -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt   -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key   -p 443:443   registry:2

singularity pull library://alpine

# Push expected to fail...
singularity push alpine_latest.sif oras://$(hostname --fqdn)/test:latest

[tri-adam]

Perhaps for the short term we could:

1. Ensure the --insecure flag to remote login honored properly for authenticated oras:// operations
2. Introduce an --insecure flag that sets insecure mode for a registry in oras operations and other pulls. This would allow insecure pulls without authenticatin.

This makes sense to me. Standardizing on --insecure makes it straightforward from a user point of view.

An --insecure flag would have slightly different semantics than the existing --http-only flag, as an insecure registry in Docker parlance allows both https with invalid certificates, and http... which it tries in that order. It might make sense to actually deprecate --http-only for consistency with other tools, and internally (see remote add --insecure). Need to consider shub, library, net sources also.

Agreed. I believe --insecure would cover the current use case for --http-only, is more descriptive, and adds consistency.

Longer term I believe we broadly favour moving away from the oras-project/oras-go client due to the mismatch of its dependencies and behavior with other push/pull functionality in Singularity. Because of the recent ORAS work in Helm there is increased interest in containers/image being able to handle non OCI image artifacts. We may be able to leverage this in future: containers/image#1370

Yes, longer term this still seems to make sense.