Proxy cache returns wrong error on missing image/tag

[MichaelOultram-pexip]

Expected behavior and actual behavior:

When using docker-py to fetch a non-existent image, it should raise docker.errors.NotFound if the image/tag is missing. This works correctly if the image is being fetched directly from DockerHub, but we get a docker.errors.APIError if fetched via harbor.

docker-py is the easiest way to demonstrate the issue, but the differences can be further shown using curl. Both harbor and dockerhub are returning a 404 error, but the body of the response is difference.

Dockerhub has:

{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"badtag"}}]}

Harbor has:

{"errors":[{"code":"NOT_FOUND","message":"resource not found: repo [docker.io/library/debian](http://docker.io/library/debian), tag badtag not found"}]}

In distribution/distribution, the MANIFEST_UNKNOWN is defined as an appropriate error code, but I do not see anything defined for NOT_FOUND. https://github.com/distribution/distribution/blob/3fe707de5c94160f40b41454e0fd0b91921d0176/registry/api/errcode/register.go#L145-L151

Steps to reproduce the problem:

• Configure a proxy cache to dockerhub on harbor. I used docker.io as the proxy cache name in my examples below

Via docker-py

>>> import docker
>>> client = docker.from_env()
>>> client.images.pull("debian:badtag")
Traceback (most recent call last):
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/api/client.py", line 265, in _raise_for_status
    response.raise_for_status()
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/images/create?tag=badtag&fromImage=debian

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/models/images.py", line 464, in pull
    pull_log = self.client.api.pull(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/api/image.py", line 429, in pull
    self._raise_for_status(response)
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/api/client.py", line 267, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.43/images/create?tag=badtag&fromImage=debian: Not Found ("manifest for debian:badtag not found: manifest unknown: manifest unknown")
>>> client.images.pull("harbor.local/docker.io/debian:badtag")
Traceback (most recent call last):
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/api/client.py", line 265, in _raise_for_status
    response.raise_for_status()
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.43/images/create?tag=badtag&fromImage=harbor.local%2Fdocker.io%2Fdebian

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/models/images.py", line 464, in pull
    pull_log = self.client.api.pull(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/api/image.py", line 429, in pull
    self._raise_for_status(response)
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/api/client.py", line 267, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/hhr7h5pab4dcy6sjbhw1xw3ym1z45h87-python3-3.11.9-env/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.43/images/create?tag=badtag&fromImage=harbor.local%2Fdocker.io%2Fdebian: Internal Server Error ("unknown: resource not found: repo docker.io/library/debian, tag badtag not found")

Via curl

Dockerhub:

#!/usr/bin/env bash
repo=library/debian
tag=badtag

token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
        | jq -r '.token')
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
curl -v -H "Accept: ${api}" -H "Accept: ${apil}" \
     -H "Authorization: Bearer $token" \
     -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}"

* Request completely sent off
< HTTP/1.1 404 Not Found
< content-type: application/json
< docker-distribution-api-version: registry/2.0
< date: Mon, 22 Jul 2024 16:34:01 GMT
< content-length: 100
< strict-transport-security: max-age=31536000
< ratelimit-limit: 100;w=21600
< ratelimit-remaining: 99;w=21600
< docker-ratelimit-source: 81.108.232.23
< 
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":"unknown tag=badtag"}]}

Harbor:

#!/usr/bin/env bash
repo=docker.io/library/debian
tag=badtag

token=$(curl -s "https://harbor.local/service/token?service=harbor-registry&scope=repository:${repo}:pull" \
        | jq -r '.token')
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
curl -v -H "Accept: ${api}" -H "Accept: ${apil}" \
     -H "Authorization: Bearer $token" \
     -s "https://harbor.local/v2/${repo}/manifests/${sha:-$tag}"

> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Mon, 22 Jul 2024 16:43:59 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 118
< Connection: keep-alive
< Set-Cookie: sid=a7d5ccf54bda3ae92612f51e6788899c; Path=/; HttpOnly
< X-Request-Id: fbb89ae1-417e-4e05-86a5-3ca9a6d9e37c
< 
{"errors":[{"code":"NOT_FOUND","message":"resource not found: repo docker.io/library/debian, tag badtag not found"}]}
* Connection #0 to host harbor.local left intact

Versions: Please specify the versions of following systems.

• harbor version: v2.11.0
• docker engine version: 27.0.3
• docker-compose version: v2.28.1

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[MichaelOultram-pexip]

Still relevant