`nerdctl push` interop issue with Harbor registry

[l2dy]

Description

Hi, I'm trying nerdctl push on a private Harbor registry with Bearer authentication and a custom token service, but the command fails with 401 Unauthorized. After further investigation, it seems to be an interoperability issue between nerdctl and Harbor registries that have disabled basic authentication.

When nerdctl requests GET /v2/<name>/blobs/<digest>, the registry returns a WWW-Authenticate header that asks the client to request for a Bearer token with the scope of repository:<name>:pull from the specified realm. nerdctl would faithfully do so and retry the request with the right credentials, so far so good.

But when nerdctl reuses the same token for POST /v2/<name>/blobs/uploads/ requests, the scope required is repository:<name>:pull,push, so this request is denied and the registry returns 401 Unauthorized. The problem is that when a request contains a Authorization header but failed to authenticate, Harbor uses Basic realm="harbor" as the authentication challenge[ref], instead of the token service configured.

The registry I'm accessing exclusively relies on Bearer tokens for authentication and has been configured to deny all requests with basic auth. Being mislead by the new WWW-Authenticate header, all subsequent requests made by nerdctl would fail.

To solve this interop issue, nerdctl could do one of the following:

1. Make a "preflight" request to the GET /v2/ API Version Check endpoint to retrieve a token without scope constraint and cache it for future requests.
2. Retry the request without an Authorization header. In this case, Harbor would return the correct WWW-Authenticate header that points to the token service.

On Harbor's side, there is https://github.com/goharbor/harbor/issues/17930, but no decision has been made yet.

Steps to reproduce the issue

1. nerdctl login ...
2. nerdctl push <image>

Describe the results you received and expected

Received: unexpected status from POST request to .../blobs/uploads/: 401 Unauthorized

Expected: Command succeed without errors.

What version of nerdctl are you using?

Client:
 Version:   v1.0.0
 OS/Arch:   linux/amd64
 Git commit:    c00780a1f5b905b09812722459c54936c9e070e6
 buildctl:
  Version:  v0.10.5
  GitCommit:    bc26045116045516ff2427201abd299043eaf8f7

Server:
 containerd:
  Version:  v1.6.8
  GitCommit:    9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:  1.1.4
  GitCommit:    5fd4c4d144137e991c4acebb2146ab1483a97925

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

Rancher Desktop for macOS

Host information

Client:
 Namespace: default
 Debug Mode:    false

Server:
 Server Version: v1.6.8
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.78-0-virt
 Operating System: Alpine Linux v3.16
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 5.809GiB
 Name: lima-rancher-desktop
 ID: 6238c1b1-ca3b-4e66-938c-d52f6c1895f5

WARNING: IPv4 forwarding is disabled
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[fahedouch]

Hi @l2dy,

Thank you for raising the issue :)

Make a "preflight" request to the GET /v2/ API Version Check endpoint to retrieve a token without scope constraint and cache it for future requests.

do you mean fetch a full-privileges token ? this seems not very safe

Retry the request without an Authorization header. In this case, Harbor would return the correct WWW-Authenticate header that points to the token service.

this may be a relevant option from the moment we are in the authentication retry loop, no need to have an expired ( useless) token.

I beleive that we should wait for an explanation from harbor side to find out the motivation behind this behavior ( that diverges from other registry already supported by nerdctl) then we can go further based on their feedback

as a workaround , you may prioritize push over pull or use an authenticate proxy to play with headers/token

[l2dy]

Make a "preflight" request to the GET /v2/ API Version Check endpoint to retrieve a token without scope constraint and cache it for future requests.

do you mean fetch a full-privileges token ? this seems not very safe

Yes, but in your threat model, does a more privileged token impose any risk?

Retry the request without an Authorization header. In this case, Harbor would return the correct WWW-Authenticate header that points to the token service.

this may be a relevant option from the moment we are in the authentication retry loop, no need to have an expired ( useless) token.

+1

I beleive that we should wait for an explanation from harbor side to find out the motivation behind this behavior ( that diverges from other registry already supported by nerdctl) then we can go further based on their feedback

No reaction on Harbor's side yet.

as a workaround , you may prioritize push over pull or use an authenticate proxy to play with headers/token

Could you elaborate on what prioritize means here? Is it to instruct clients to fetch a write-scoped token for read-only APIs, to give a more privileged token than what the client requested, or for nerdctl to attempt write-scoped APIs first in the push command?

Edit: typo

[l2dy]

@fahedouch Any update on this? Should we go with dropping Authorization header on retry failure?

[fahedouch]

@fahedouch Any update on this? Should we go with dropping Authorization header on retry failure?

SGTM