cmoulliard
commented
3 years ago Here is more information abut the problem
How can I fix such an issue which occurs when I try to pull an image (or when a pod tries to pull the image) from a TLS local container registry
root@kind-control-plane:/# crictl pull localhost:5000/hello-app
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "localhost:5000/hello-app:latest": failed to resolve reference "localhost:5000/hello-app:latest": failed to do request: Head "http://kind-registry:5000/v2/hello-app/manifests/latest?ns=localhost%3A5000": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02" Log of the registry reports such an error
2021/09/28 08:22:05 http: TLS handshake error from 172.19.0.2:52546: tls: first record does not look like a TLS handshakeI can of course curl the docker registry from a container
curl -vk -u admin:snowdrop https://kind-registry:5000/v2/hello-app/manifests/latest?ns=localhost%3A5000
* Trying fc00:f853:ccd:e793::3:5000...
* Connected to kind-registry (fc00:f853:ccd:e793::3) port 5000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=BE; ST=Namur; L=Florennes; O=Red Hat; OU=Snowdrop; CN=localhost
* start date: Sep 28 07:01:23 2021 GMT
* expire date: Sep 28 07:01:23 2022 GMT
* issuer: C=BE; ST=Namur; L=Florennes; O=Red Hat; OU=Snowdrop; CN=localhost
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Basic with user 'admin'
* Using Stream ID: 1 (easy handle 0x55f8e6e0b890)
> GET /v2/hello-app/manifests/latest?ns=localhost%3A5000 HTTP/2
> Host: kind-registry:5000
> authorization: Basic YWRtaW46c25vd2Ryb3A=
> user-agent: curl/7.74.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 404
< content-type: application/json; charset=utf-8
< docker-distribution-api-version: registry/2.0
< x-content-type-options: nosniff
< content-length: 96
< date: Tue, 28 Sep 2021 07:27:25 GMT
<
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"latest"}}]}
* Connection #0 to host kind-registry left intactcontainerd config is
root@kind-control-plane:/# cat /etc/containerd/config.toml
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
restrict_oom_score_adj = false
sandbox_image = "k8s.gcr.io/pause:3.5"
tolerate_missing_hugepages_controller = true
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true
snapshotter = "overlayfs"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."localhost:5000"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."localhost:5000".tls]
cert_file = "/etc/docker/certs.d/localhost/localhost.crt"
key_file = "/etc/docker/certs.d/localhost/localhost.key"
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
endpoint = ["http://kind-registry:5000"]Here is what crictl info reports
"registry": {
"configPath": "",
"mirrors": {
"localhost:5000": {
"endpoint": [
"https://kind-registry:5000"
]
}
},
"configs": {
"localhost:5000": {
"auth": null,
"tls": {
"insecure_skip_verify": true,
"caFile": "",
"certFile": "",
"keyFile": ""
}
}
},
"auths": null,
"headers": null
},
Issue
The following push fails against a localhost:5000 registry running as container
Bash script used