I upgrade Harbor from v 2.8.4 to v2.10.2 with harbor-helm today, and I got some errors when testing:
docker login failed with 401 Unauthorized
OIDC user login failed with 401 Unauthorized
robot account login failed with 401 Unauthorized
fail to pull images with 401 Unauthorized
Some logs shown:
docker login errors:
docker login https://harbor.xxx
Username: robot$harborupgradetest
Password:
Error response from daemon: login attempt to https://harbor.xxx/v2/ failed with status: 401 Unauthorized
harbor-core logs:
2024-06-19T15:15:03Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="03e33ec8faa7014fed6be743a91ce4a0" traceID="b83be82217786d316bb066b1966d9e17"]: failed to verify secret, username: Jinshuai_Ni, error: failed to refresh token, username: Jinshuai_Ni, error: oauth2: "invalid_grant" "Offline user session not found"
2024-06-19T15:15:03Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.9.62.130" requestID="03e33ec8faa7014fed6be743a91ce4a0" traceID="b83be82217786d316bb066b1966d9e17" user agent="containerd/1.7.11"]: failed to authenticate user:Jinshuai_Ni, error:not supported
2024-06-19T15:15:04Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="e86dfdd32ed638e89d3250033bad21fd" traceID="b83be82217786d316bb066b1966d9e17"]: failed to verify secret, username: Jinshuai_Ni, error: failed to refresh token, username: Jinshuai_Ni, error: oauth2: "invalid_grant" "Offline user session not found"
2024-06-19T15:15:04Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.9.62.130" requestID="e86dfdd32ed638e89d3250033bad21fd" traceID="b83be82217786d316bb066b1966d9e17" user agent="containerd/1.7.11"]: failed to authenticate user:Jinshuai_Ni, error:not supported
2024-06-19T15:15:17Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="b47acdf59c6eb432f6cab4fa08237a3f"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2024-06-19T15:15:17Z [ERROR] [/pkg/reg/adapter/native/adapter.go:126]: failed to ping registry https://xxx: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
2024-06-19T15:15:58Z [INFO] [/server/middleware/security/robot.go:71][requestID="9a3e1c823caf91df5520dcdb28ba4d21" traceID="66af0e823e57929c948cfbb46e4bb10e"]: a robot security context generated for request HEAD /v2/iescapital-cloud/capital_server_2024/manifests/dev.2408_599
2024-06-19T15:15:58Z [INFO] [/server/middleware/security/robot.go:71][requestID="f84402ec8942dc719818a1599b9cfa80" traceID="66af0e823e57929c948cfbb46e4bb10e"]: a robot security context generated for request HEAD /v2/iescapital-cloud/capital_server_2024/manifests/dev.2408_599
2024-06-19T15:16:35Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="62ff0e8632a44b44ffc13fa79b21c165"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2024-06-20T03:07:55Z [ERROR] [/server/middleware/security/oidc_cli.go:68][requestID="334fcd11abfd96644d485b9ab971f268"]: failed to verify secret, username: robot, error: failed to get oidc user info, error: <QuerySeter> no row found
2024-06-20T03:07:55Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.27.63.71" requestID="334fcd11abfd96644d485b9ab971f268" user agent="docker/1.13.1 go/go1.10.3 kernel/5.10.16.3-microsoft-standard-WSL2 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \(linux\))"]: failed to authenticate user:robot, error:not supported
pod logs:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 79s default-scheduler Successfully assigned https://xxx/cis-worker-capx2408-5555c985bf-s28jb to ip-xxx.ap-northeast-1.compute.internal
Normal Pulling 35s (x3 over 79s) kubelet Pulling image "https://xxx/xxx/supporttools:capitalboto3"
Warning Failed 34s (x3 over 78s) kubelet Failed to pull image "https://xxx/xxx/supporttools:capitalboto3": failed to pull and unpack image "https://xxx/xxx/supporttools:capitalboto3": failed to resolve reference "https://xxx/xxx/supporttools:capitalboto3": unexpected status from HEAD request to https://xxx/v2/xxx/supporttools/manifests/capitalboto3: 401 Unauthorized
Warning Failed 34s (x3 over 78s) kubelet Error: ErrImagePull
Normal BackOff 4s (x4 over 78s) kubelet Back-off pulling image "https://xxx/xxx/supporttools:capitalboto3"
Warning Failed 4s (x4 over 78s) kubelet Error: ImagePullBackOff
another error logs found when I tried to re-deploy harbor-helm with the same version:
And no errors when docker login with robot account:
2024-06-20T07:27:23Z [INFO] [/server/middleware/security/robot.go:71][requestID="9930899d4c802ed0ad94be0f6a9a9308"]: a robot security context generated for request GET /service/token
2024-06-20T07:28:02Z [INFO] [/server/middleware/security/robot.go:71][requestID="58ef3c80168928bb8a6f35787f4e1da2"]: a robot security context generated for request GET /service/token
In my situation I can login Harbor UI with OIDC user, and I can see/search images in projects as expected. Robot account can be created/removed/edited, but seems all operations with API calls would fail. I have searched quite a lot in issues but can not find the root cause. Can anyone Senior can help me on this? Thanks quite a lot for your help and time here.
Hi team,
I upgrade Harbor from v 2.8.4 to v2.10.2 with harbor-helm today, and I got some errors when testing:
Some logs shown:
docker login errors:
harbor-core logs:
pod logs:
another error logs found when I tried to re-deploy harbor-helm with the same version:
And no errors when docker login with robot account:
In my situation I can login Harbor UI with OIDC user, and I can see/search images in projects as expected. Robot account can be created/removed/edited, but seems all operations with API calls would fail. I have searched quite a lot in issues but can not find the root cause. Can anyone Senior can help me on this? Thanks quite a lot for your help and time here.