Nydus-Snapshotter Auth Error

Hi Team,

I’m currently integrating Nydus with Dragonfly in a Kubernetes cluster installed via Helm charts. While Dragonfly works seamlessly without Nydus, I’m encountering authentication issues when the two are integrated. The nydusd is running as a daemonset.

image: ghcr.io/containerd/nydus-snapshotter tag: v0.9.0

Helm Chart: https://github.com/dragonflyoss/helm-charts/blob/main/INSTALL.md

Nydus-Config:

{ "device": { "backend": { "type": "registry", "config": { "mirrors": [{"auth_through":false,"headers":{"X-Dragonfly-Registry":"http://harbor-registry.com/"},"host":"http://127.0.0.1:65001/","ping_url":"http://127.0.0.1:40901/server/ping"}], "scheme": "http", "skip_verify": true, "timeout": 5, "connect_timeout": 5, "retry_limit": 5 } }, "cache": { "type": "blobcache", "config": { "work_dir": "/var/lib/nydus/cache/" } } }, "mode": "direct", "digest_validate": false, "iostats_files": false, "enable_xattr": true, "fs_prefetch": { "enable": true, "threads_count": 8, "merging_size": 1048576, "prefetch_all": true } }

ContainerD Config:

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["https://mirror.gcr.io/", "https://registry-1.docker.io/"] [proxy_plugins] [proxy_plugins.nydus] type = "snapshot" address = "/run/containerd-nydus/containerd-nydus-grpc.sock" [plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor-registry.com"] endpoint = ["http://127.0.0.1:65001/","http://harbor-registry.com/"] [plugins."io.containerd.grpc.v1.cri".registry.configs."127.0.0.1:65001".auth] username = "username" password = "mypassword" auth = "" identitytoken = ""

Error Message: Nydus log:

Read from backend failed: Registry(Request(ErrorWithMsg("401 Unauthorized"))), retry count 10 [2023-11-15 12:08:22.991641 +00:00] INFO [storage/src/cache/cachedfile.rs:74] temporarily failed to get blob.meta, I/O error (os error 5) [2023-11-15 12:08:23.003567 +00:00] ERROR [error/src/error.rs:21] Error: "failed to read metadata from backend(compressor is none), Registry(Request(ErrorWithMsg(\"401 Unauthorized\")))" at storage/src/meta/mod.rs:659 note: enable RUST_BACKTRACE=1 env to display a backtrace

Pod Event:

Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount28632954: read /var/lib/containerd/tmpmounts/containerd-mount28632954/etc/passwd: invalid argument

I have also tried doing the auth via the kubernetes service account by following the below doc: https://github.com/containerd/nydus-snapshotter/blob/main/docs/configure_nydus.md#use-serviceaccount

Established a service account, configured registry authentication, and seamlessly applied the authentication in a DaemonSet through imagePullSecrets.

containerd / nydus-snapshotter

Nydus-Snapshotter Auth Error #557