search

containers / buildah

A tool that facilitates building OCI images.
https://buildah.io
Apache License 2.0
7.46k stars 786 forks source link

buildah times out trying HTTPS before trying HTTP for insecure registries #5531

Open mrled opened 6 months ago

mrled commented 6 months ago

Description

When a registry is configured as insecure in /etc/containers/registries.conf, buildah first tries https:// and waits a full 30 seconds for it to time out before trying HTTP.

Steps to reproduce the issue:

  1. Configure a registry as insecure in /etc/containers/registries.conf
  2. Run a command like buildah --debug pull registry.registry.svc.cluster.local/repository/clustergit:latest2

Describe the results you received:

In this debug output, the newlines in the middle are where it paused for 30 seconds.

(The final error, about "manifest unknown", is expected in this case. The only issue I'm reporting is the HTTPS timeout before trying HTTP, not the unknown image.)

~ # time buildah --debug pull registry.registry.svc.cluster.local/repository/clustergit:latest2
DEBU[0000] effective capabilities: [audit_control=false audit_read=false audit_write=true block_suspend=false bpf=false checkpoint_restore=false chown=true dac_override=true dac_read_search=false fowner=true fsetid=true ipc_lock=false ipc_owner=false kill=true lease=false linux_immutable=false mac_admin=false mac_override=false mknod=true net_admin=false net_bind_service=true net_broadcast=false net_raw=true perfmon=false setfcap=true setgid=true setpcap=true setuid=true sys_admin=false sys_boot=false sys_chroot=true sys_module=false sys_nice=false sys_pacct=false sys_ptrace=false sys_rawio=false sys_resource=false sys_time=false sys_tty_config=false syslog=false wake_alarm=false] 
DEBU[0000] Running [buildah-in-a-user-namespace --debug pull registry.registry.svc.cluster.local/repository/clustergit:latest2] with environment [KUBERNETES_SERVICE_PORT=443 KUBERNETES_PORT=tcp://10.96.0.1:443 HOSTNAME=tmpalpine SHLVL=1 HOME=/root OLDPWD=/ TERM=xterm KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 KUBERNETES_SERVICE_HOST=10.96.0.1 PWD=/root TMPDIR=/var/tmp _CONTAINERS_USERNS_CONFIGURED=1 BUILDAH_ISOLATION=rootless], UID map [{ContainerID:0 HostID:0 Size:4294967295}], and GID map [{ContainerID:0 HostID:0 Size:4294967295}] 
DEBU[0000] effective capabilities: [audit_control=true audit_read=true audit_write=true block_suspend=true bpf=true checkpoint_restore=true chown=true dac_override=true dac_read_search=true fowner=true fsetid=true ipc_lock=true ipc_owner=true kill=true lease=true linux_immutable=true mac_admin=true mac_override=true mknod=true net_admin=true net_bind_service=true net_broadcast=true net_raw=true perfmon=true setfcap=true setgid=true setpcap=true setuid=true sys_admin=true sys_boot=true sys_chroot=true sys_module=true sys_nice=true sys_pacct=true sys_ptrace=true sys_rawio=true sys_resource=true sys_time=true sys_tty_config=true syslog=true wake_alarm=true] 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: storage already configured with a mount-program 
DEBU[0000] backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Pulling image registry.registry.svc.cluster.local/repository/clustergit:latest2 (policy: missing) 
DEBU[0000] Looking up image "registry.registry.svc.cluster.local/repository/clustergit:latest2" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "registry.registry.svc.cluster.local/repository/clustergit:latest2" ... 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]registry.registry.svc.cluster.local/repository/clustergit:latest2" does not resolve to an image ID 
DEBU[0000] Trying "registry.registry.svc.cluster.local/repository/clustergit:latest2" ... 
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]registry.registry.svc.cluster.local/repository/clustergit:latest2" does not resolve to an image ID 
DEBU[0000] Trying "registry.registry.svc.cluster.local/repository/clustergit:latest2" ... 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/00-shortnames.conf" 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate registry.registry.svc.cluster.local/repository/clustergit:latest2 for registry.registry.svc.cluster.local/repository/clustergit:latest2 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]registry.registry.svc.cluster.local/repository/clustergit:latest2" 
Trying to pull registry.registry.svc.cluster.local/repository/clustergit:latest2...
DEBU[0000] Copying source image //registry.registry.svc.cluster.local/repository/clustergit:latest2 to destination image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]registry.registry.svc.cluster.local/repository/clustergit:latest2 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "registry.registry.svc.cluster.local/repository/clustergit:latest2" 
DEBU[0000] Found credentials for registry.registry.svc.cluster.local/repository/clustergit in credential helper containers-auth.json in file /var/tmp/containers-user-0/containers/containers/auth.json 
DEBU[0000]  No signature storage configuration found for registry.registry.svc.cluster.local/repository/clustergit:latest2, using built-in default file:///var/lib/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.registry.svc.cluster.local 
DEBU[0000] GET https://registry.registry.svc.cluster.local/v2/ 

DEBU[0030] Ping https://registry.registry.svc.cluster.local/v2/ err Get "https://registry.registry.svc.cluster.local/v2/": dial tcp 10.109.158.230:443: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.registry.svc.cluster.local/v2/", Err:(*net.OpError)(0xc0000a42d0)}) 
DEBU[0030] GET http://registry.registry.svc.cluster.local/v2/ 
DEBU[0030] Ping http://registry.registry.svc.cluster.local/v2/ status 401 
DEBU[0030] GET http://auth.registry.svc.cluster.local/auth?account=browser&scope=repository%3Arepository%2Fclustergit%3Apull&service=registry.younix.us 
DEBU[0030] Increasing token expiration to: 60 seconds   
DEBU[0030] GET http://registry.registry.svc.cluster.local/v2/repository/clustergit/manifests/latest2 
DEBU[0030] Content-Type from manifest GET is "application/json; charset=utf-8" 
DEBU[0030] Accessing "registry.registry.svc.cluster.local/repository/clustergit:latest2" failed: reading manifest latest2 in registry.registry.svc.cluster.local/repository/clustergit: manifest unknown 
DEBU[0030] Error pulling candidate registry.registry.svc.cluster.local/repository/clustergit:latest2: initializing source docker://registry.registry.svc.cluster.local/repository/clustergit:latest2: reading manifest latest2 in registry.registry.svc.cluster.local/repository/clustergit: manifest unknown 
Error: initializing source docker://registry.registry.svc.cluster.local/repository/clustergit:latest2: reading manifest latest2 in registry.registry.svc.cluster.local/repository/clustergit: manifest unknown
DEBU[0030] shutting down the store                      
DEBU[0030] exit status 125                              
Command exited with non-zero status 125
real    0m 30.10s
user    0m 0.02s
sys 0m 0.01s

Describe the results you expected:

It should try HTTP immediately, since it's configured as an insecure registry.

Output of rpm -q buildah or apt list buildah:

I'm on Alpine, so:

~ # apk list -i | grep buildah
buildah-1.33.6-r3 x86_64 {buildah} (Apache-2.0) [installed]

Output of buildah version:

~ # buildah version
Version:         1.33.6
Go Version:      go1.21.10
Image Spec:      1.1.0-rc.5
Runtime Spec:    1.1.0
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.29.2
Git Commit:      1.33.6
Built:           Sun May 12 07:25:43 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of podman version if reporting a podman build issue:

This happens with podman too, for what it's worth.

~ # podman version
Client:       Podman Engine
Version:      4.8.3
API Version:  4.8.3
Go Version:   go1.21.10
Built:        Sun May 12 07:25:43 2024
OS/Arch:      linux/amd64

*Output of `cat /etc/release`:**

~ # cat /etc/*release
3.19.1
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19.1
PRETTY_NAME="Alpine Linux v3.19"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

Output of uname -a:

Linux tmpalpine 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Apr 4 18:13:02 UTC 2024 x86_64 Linux

Output of cat /etc/containers/storage.conf:

~ # grep -v '^#' /etc/containers/storage.conf  | grep -v '^$'
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}
[storage.options.overlay]
mountopt = "nodev"
[storage.options.thinpool]
github-actions[bot] commented 5 months ago

A friendly reminder that this issue had no activity for 30 days.

thomascube commented 6 days ago

I'm having a similar situation where the docker registry used with --cache-from and --cache-to runs in the local network with http://. Is it possible to disable the ping to https:// with an option? In my case the check for each layer wastes 30 seconds until the https ping runs into a timeout.

thomascube commented 5 days ago

I also tried with BUILD_REGISTRY_SOURCES='{"insecureRegistries":["docker-registry.namespace.svc.cluster.local"]}' but without success.

$ buildah version
Version:         1.31.3
Go Version:      go1.20.12
Image Spec:      1.1.0-rc.3
Runtime Spec:    1.1.0-rc.3
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.26.2
Git Commit:
Built:           Wed Mar 13 11:28:44 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64
thomascube commented 5 days ago

Extract from the buildah debug log:

time="2024-11-19T08:11:37Z" level=debug msg="Copying source image //my-docker-registry.demo-dev-tekton.svc.cluster.local/cache:01ed03a741b46d77ed009d903877d5f8e33d858e4602d80530905a2dd0a1d28c to destination image [vfs@/home/build/.local/share/containers/storage+/build-tmp/containers-user-1000/containers]my-docker-registry.demo-dev-tekton.svc.cluster.local/cache:01ed03a741b46d77ed009d903877d5f8e33d858e4602d80530905a2dd0a1d28c"
time="2024-11-19T08:11:37Z" level=debug msg="BUILD_REGISTRY_SOURCES set \"{\\\"insecureRegistries\\\":[\\\"my-docker-registry.demo-dev-tekton.svc.cluster.local\\\"]}\""
time="2024-11-19T08:11:37Z" level=debug msg="BUILD_REGISTRY_SOURCES set \"{\\\"insecureRegistries\\\":[\\\"my-docker-registry.demo-dev-tekton.svc.cluster.local\\\"]}\""
time="2024-11-19T08:11:37Z" level=debug msg="Using registries.d directory /etc/containers/registries.d"
time="2024-11-19T08:11:37Z" level=debug msg="Trying to access \"my-docker-registry.demo-dev-tekton.svc.cluster.local/cache:01ed03a741b46d77ed009d903877d5f8e33d858e4602d80530905a2dd0a1d28c\""
time="2024-11-19T08:11:37Z" level=debug msg="No credentials matching my-docker-registry.demo-dev-tekton.svc.cluster.local/cache found in /build-tmp/containers-user-1000/containers/containers/auth.json"
time="2024-11-19T08:11:37Z" level=debug msg="No credentials matching my-docker-registry.demo-dev-tekton.svc.cluster.local/cache found in /home/build/.config/containers/auth.json"
time="2024-11-19T08:11:37Z" level=debug msg="No credentials matching my-docker-registry.demo-dev-tekton.svc.cluster.local/cache found in /home/build/.docker/config.json"
time="2024-11-19T08:11:37Z" level=debug msg="No credentials matching my-docker-registry.demo-dev-tekton.svc.cluster.local/cache found in /home/build/.dockercfg"
time="2024-11-19T08:11:37Z" level=debug msg="No credentials for my-docker-registry.demo-dev-tekton.svc.cluster.local/cache found"
time="2024-11-19T08:11:37Z" level=debug msg=" No signature storage configuration found for my-docker-registry.demo-dev-tekton.svc.cluster.local/cache:01ed03a741b46d77ed009d903877d5f8e33d858e4602d80530905a2dd0a1d28c, using built-in default file:///home/build/.local/share/containers/sigstore"
time="2024-11-19T08:11:37Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/my-docker-registry.demo-dev-tekton.svc.cluster.local"
time="2024-11-19T08:11:37Z" level=debug msg="GET https://my-docker-registry.demo-dev-tekton.svc.cluster.local/v2/"
time="2024-11-19T08:12:07Z" level=debug msg="Ping https://my-docker-registry.demo-dev-tekton.svc.cluster.local/v2/ err Get \"https://my-docker-registry.demo-dev-tekton.svc.cluster.local/v2/\": dial tcp 198.18.53.65:443: i/o timeout (&url.Error{Op:\"Get\", URL:\"https://my-docker-registry.demo-dev-tekton.svc.cluster.local/v2/\", Err:(*net.OpError)(0xc001494000)})"
time="2024-11-19T08:12:07Z" level=debug msg="GET http://my-docker-registry.demo-dev-tekton.svc.cluster.local/v2/"
time="2024-11-19T08:12:07Z" level=debug msg="Ping http://my-docker-registry.demo-dev-tekton.svc.cluster.local/v2/ status 200"
time="2024-11-19T08:12:07Z" level=debug msg="GET http://my-docker-registry.demo-dev-tekton.svc.cluster.local/v2/cache/manifests/01ed03a741b46d77ed009d903877d5f8e33d858e4602d80530905a2dd0a1d28c"
time="2024-11-19T08:12:07Z" level=debug msg="Content-Type from manifest GET is \"application/vnd.oci.image.manifest.v1+json\""