k3os v0.19.5-rc.1 does not run pod from private registry. Error is x509: certificate signed by unknown authority

[gbonazzoli]

Version (k3OS / kernel) k3os version v0.19.5-rc.1 5.4.0-54-generic #60 SMP Fri Nov 20 17:09:10 UTC 2020

Architecture x86_64

Describe the bug k3os v0.19.5-rc.1 does not run pod from private registry. Error is x509: certificate signed by unknown authority

I've configured each node of the cluster with the following /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

` [plugins.opt] path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri] stream_server_address = "127.0.0.1" stream_server_port = "10010" enable_selinux = false sandbox_image = "docker.io/rancher/pause:3.1"

[plugins.cri.registry] [plugins.cri.registry.mirrors] [plugins.cri.registry.mirrors."docker.io"] endpoint = ["https://registry-1.docker.io"] [plugins.cri.registry.mirrors."registry.poltronesofa.com"] endpoint = ["https://registry.poltronesofa.com"]

[plugins.cri.registry.configs."registry.poltronesofa.com".tls] cert_file = "/opt/wildcard_pes_ok.crt" key_file = "/opt/wildcard_pes_ok.key" ca_file = "/opt/RapidSSLTLSDVRSAMixedSHA2562020CA-1.crt"

[plugins.cri.containerd] snapshotter = "overlayfs"

[plugins.cri.cni] bin_dir = "/var/lib/rancher/k3s/data/b9574be94e4edbdbb93a39a2cb1f4e4df3ba699171a8b86863d1e8c421c91f63/bin" conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" `

To Reproduce The error arises when trying to run a simple pod. Write a simple pippo.yaml

` apiVersion: v1 kind: Pod metadata: name: pippo spec: containers:

• name: hello-world image: registry.poltronesofa.com/hello-world:latest imagePullSecrets:
• name: regcred `

apply it with kubectl apply -f pippo.yaml then describe it. You will have:

`kubectl describe pod pippo

Type Reason Age From Message

---

Normal Scheduled 9m55s default-scheduler Successfully assigned default/pippo to k3os-node02 Normal Pulling 8m22s (x4 over 9m55s) kubelet Pulling image "registry.poltronesofa.com/hello-world:latest" Warning Failed 8m22s (x4 over 9m55s) kubelet Failed to pull image "registry.poltronesofa.com/hello-world:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry.poltronesofa.com/hello-world:latest": failed to resolve reference "registry.poltronesofa.com/hello-world:latest": failed to do request: Head "https://registry.poltronesofa.com/v2/hello-world/manifests/latest": x509: certificate signed by unknown authority Warning Failed 8m22s (x4 over 9m55s) kubelet Error: ErrImagePull Warning Failed 8m9s (x6 over 9m54s) kubelet Error: ImagePullBackOff Normal BackOff 4m48s (x20 over 9m54s) kubelet Back-off pulling image "registry.poltronesofa.com/hello-world:latest" `

[dweomer]

x509: certificate signed by unknown authority

This is telling you that your private registry certificate is not signed by anything in the system's trust store at /etc/ssl/cert.pem. One way to rectify this is to drop the cert for your private registry (or more ideally, the cert that signed this cert) into /usr/local/share/ca-certificates/my-private-registry.pem via a write_files directive in your config.yaml with a corresponding run_cmd entry that invokes update-ca-certificates. This will update your system at every boot to trust any outbound connections to your private registry.

[jamieg-sixworks]

I'm also having this issue - I'm not sure I can your /usr/.. method above because I don't think usr is writable? More frustratingly, the "insecure_skip_verify" switch in the k3s registry configuration doesn't seem to be honoured.

I've confirmed that I can do the pull from each K3os node as long as I specific "-k" in the "ctr image pull" manually. "sudo crictl info" shows the expected "insecure_skip_verify": true

[dweomer]

/usr/local is a symlink into persistent, read-write disk.

/etc is ephemeral and writable.

[jamieg-sixworks]

/usr/local is a symlink into persistent, read-write disk.

/etc is ephemeral and writable.

Thanks - creating /var/local/share/ca-certificates and following this method worked for me. Not sure why the "insecure_skip_verify": true isn't doing what i expect.