Buildah can't pull from index.docker.io, but can pull from docker.io

[johnmcollier]

Description

We're running Buildah on Kubernetes and we're finding that buildah pull index.docker.io/<some-image> fails, while buildah pull docker.io/<some-image> doesn't fail.

[[root@codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr /]# buildah pull index.docker.io/appsody/nodejs-express:0.2
The following failures happened while trying to pull image specified by "index.docker.io/appsody/nodejs-express:0.2" based on search registries in /etc/containers/registries.conf:
* "localhost/index.docker.io/appsody/nodejs-express:0.2": Error initializing source docker://localhost/index.docker.io/appsody/nodejs-express:0.2: pinging docker registry returned: Get https://localhost/v2/: dial tcp [::1]:443: connect: connection refused
* "registry.access.redhat.com/index.docker.io/appsody/nodejs-express:0.2": Error initializing source docker://registry.access.redhat.com/index.docker.io/appsody/nodejs-express:0.2: Error reading manifest 0.2 in registry.access.redhat.com/index.docker.io/appsody/nodejs-express: name unknown: Repo not found
* "docker.io/index.docker.io/appsody/nodejs-express:0.2": Error initializing source docker://index.docker.io/appsody/nodejs-express:0.2: Error reading manifest 0.2 in docker.io/index.docker.io/appsody/nodejs-express: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

* "registry.fedoraproject.org/index.docker.io/appsody/nodejs-express:0.2": Error initializing source docker://registry.fedoraproject.org/index.docker.io/appsody/nodejs-express:0.2: Error reading manifest 0.2 in registry.fedoraproject.org/index.docker.io/appsody/nodejs-express: manifest unknown: manifest unknown
* "quay.io/index.docker.io/appsody/nodejs-express:0.2": Error initializing source docker://quay.io/index.docker.io/appsody/nodejs-express:0.2: unexpected http code: 400 (Bad Request), URL: https://quay.io/v2/auth?scope=repository%3Aindex.docker.io%2Fappsody%2Fnodejs-express%3Apull&service=quay.io
* "registry.centos.org/index.docker.io/appsody/nodejs-express:0.2": Error initializing source docker://registry.centos.org/index.docker.io/appsody/nodejs-express:0.2: Error reading manifest 0.2 in registry.centos.org/index.docker.io/appsody/nodejs-express: manifest unknown: manifest unknown

[root@codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr /]# buildah pull docker.io/appsody/nodejs-express:0.2
Getting image source signatures
Copying blob e11ff976ff71 skipped: already exists
Copying blob e6cb98e32a52 skipped: already exists
Copying blob 42cfa3699b05 skipped: already exists
Copying blob ae1b8d879bad skipped: already exists
Copying blob 9cc2ad81d40d skipped: already exists
Copying blob 753f76cbf7de skipped: already exists
Copying blob 482204c4ceb2 skipped: already exists
Copying blob 710b6060865b skipped: already exists
Copying blob ce1774bcd326 skipped: already exists
Copying blob 053cac798c4e skipped: already exists
Copying blob 20165ca6bc3c skipped: already exists
Copying blob 9ab8d1383a03 skipped: already exists
Copying blob e803a4ef44e2 skipped: already exists
Copying blob 70cad6f7dde4 done
Copying blob afdc6edfe326 done
Copying blob 3ac5937a03f4 done
Copying blob 72e9ce552cca done
Copying config 825a2382a4 done
Writing manifest to image destination
Storing signatures

In the logs I see:

denied: requested access to the resource is denied
unauthorized: authentication required

but it's a public repository and shouldn't need authentication (and if we use docker.io/... it's fine).

Steps to reproduce the issue:

1. Run buildah on Kubernetes
2. Run buildah pull index.docker.io/appsody/nodejs-express:0.2. It will fail and show the error I posted above.
3. Run buildah pull docker.io/appsody/nodejs-express:0.2. It will pull the image fine.

Describe the results you received: Buildah fails to pull from `index.docker.io/appsody/nodejs-express:0.2

Describe the results you expected: Buildah fails can pull from `index.docker.io/appsody/nodejs-express:0.2

Output of rpm -q buildah or apt list buildah:

[root@codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr /]# rpm -q buildah
buildah-1.11.2-2.git0bafbfe.el7.x86_64

Output of buildah version:

[root@codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr /]# buildah version
Version:         1.11.2
Go Version:      go1.10.2
Image Spec:      1.0.1
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:
Git Commit:
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

*Output of `cat /etc/release`:**

[root@codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr /]# cat /etc/*release
CentOS Linux release 7.6.1810 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)

Output of uname -a:

[root@codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr /]# uname -a
Linux codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr 4.18.0-80.11.2.el8_0.x86_64 #1 SMP Sun Sep 15 11:24:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[root@codewind-workspacewxe4fco4pwuzb6y1-59546655-sq6zr /]# cat /etc/containers/storage.conf
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base
# device.
# mkfsarg = ""

# mountopt specifies extra mount options used when mounting the thin devices.
# mountopt = ""

# use_deferred_removal Marking device for deferred removal
# use_deferred_removal = "True"

# use_deferred_deletion Marking device for deferred deletion
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

[mtrmac]

Why is that expected to work at all? The official name of the registry is just docker.io; the underlying hostnames are not a part of the UI contract, to my knowledge.

---

And even if you did want to use the underlying host names for some reason, index.docker.io is the V1 (GitHub.com/docker/registry) server hostname; the V2 (GitHub.com/docker/distribution) host is registry-1.docker.io. containers/image has never supported the V1 protocol.

Given that, while index.docker.io does reply to the V2 API, it’s not set up consistently to do that

$ curl -iL https://index.docker.io/v2/
…
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
…

so, the client successfully obtains a token for registry.docker.io, but the server at index.docker.io refuses to accept it.

I suppose it would be cleaner if the index.docker.io server just refused to accept the /v2/ paths; in principle it would be might be nice to implement V1 registry protocol in containers/image, but that’s extremely unlikely to happen.

Ultimately, the end-user recommendation is the same either way: don’t do that.

[mtrmac]

in principle it would be might be nice to implement V1 registry protocol in containers/image, but that’s extremely unlikely to happen.

(And even if it did happen, c/image would certainly try V2 before V1, run into the inconsistent V2 configuration and not fall back to V1. So the only way it could ultimately work is by editing the index.docker.io registry name back to docker.io, the way your workaround does — but I don’t see any reason to expect that index.docker.io references should work, so far; they just look like mistakes in the caller/input that should be fixed in the caller/input.)

[vrothberg]

Just a drive by comment as it popped at the top of my mailbox. Docker allows to pull from index.docker.io and more things. See https://github.com/moby/moby/pull/34319#discussion_r162021213 for an example.

[vrothberg]

All inputs below will yield the same image (plenty of hard-coded magic):

docker pull busybox
docker pull library/busybox
docker pull docker.io/busybox
docker pull docker.io/library/busybox
docker pull index.docker.io/busybox
docker pull index.docker.io/library/busybox
docker pull registry-1.docker.io/library/busybox

[mtrmac]

In there, I see https://github.com/moby/moby/pull/34319#discussion_r167745783 :

This is mixing hostnames and image names. Images don't have names like registry-1.docker.io or index.docker.io and should not be considered "official". The "official" namespace is docker.io.