Enabling OIDC causes all robot accounts to generate error: failed to verify secret, username: robot$gitlab, error: failed to get oidc user info, error: <QuerySeter> no row

[sharkymcdongles]

It appears when OIDC is enabled the auth goes through OIDC even if the account is a robot account with no OIDC. This causes the logs to be inundated with this error message:

2021-07-02T14:55:11Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="a2af154dd57eca5e5a05a8ac4012a311"]: failed to verify secret, username: robot$gitlab, error: failed to get oidc user info, error: <QuerySeter> no row

Since robot accounts cannot even be linked to OIDC accounts is there some sort of way to stop these log messages and errors?

I am using v2.3.0. OIDC settings are:

OIDC Provider Name : azuread
OIDC Endpoint: https://login.microsoftonline.com/CENSORED/v2.0
OIDC Client ID: CENSORED
OIDC Client Secret: CENSORED
Group Claim Name: groups
OIDC Admin Group: CENSORED
OIDC Scope: openid,email,profile,offline_access
Verify Certificate: On
Automatic Onboarding: On
Username Claim: email

[DennisGlindhart]

I'm experiencing the same on a cleanly installed 2.3. Pretty much same settings as OP. (+ Also CLI for secret/login from docker doesn't seem to work either - Don't know if related, but could be something general in "auth-selection" logic when OIDC is enabled. )

[sharkymcdongles]

I also get random failures where the authorisation header is empty on docker auth to registry that seem like they might be related and sound similar to what Dennis wrote. Perhaps they are one and the same.

On Fri, 2 Jul 2021, 23:19 Dennis Glindhart, @.***> wrote:

I'm experiencing the same on a cleanly installed 2.3. Pretty much same settings as OP. (+ Also CLI for secret/login from docker doesn't seem to work either - Don't know if related, but could be something general in "auth-selection" logic when OIDC is enabled. )

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/15253#issuecomment-873261404, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6KEUKPGU7U2PKZQEHXQ7TTVYUMJANCNFSM47W4E4LQ .

[DennisGlindhart]

Can reproduce on clean version 2.2.2 also, so nothing new for 2.3

[sharkymcdongles]

Some more info from debug logs:

2021-07-03T11:37:41Z [DEBUG] [/pkg/oidc/secret.go:75]: Verifying the secret for user: robot$gitlab
2021-07-03T11:37:41Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="1135d5b3c15180380de126e5d2617cc9"]: failed to verify secret, username: robot$gitlab, error: failed to get oidc user info, error: <QuerySeter> no row found
2021-07-03T11:37:41Z [ERROR] [/pkg/token/token.go:66]: parse token error, token contains an invalid number of segments
2021-07-03T11:37:41Z [DEBUG] [/server/middleware/security/robot.go:49][requestID="1135d5b3c15180380de126e5d2617cc9"]: failed to decrypt robot token of v1 robot: robot$gitlab, as: token contains an invalid number of segments

Seems it tries to verify the robot token via oidc despite it not being oidc and then it fails to do so and then the token it generates is invalid.

Then we get these errors that cause calls to fail to the registry resulting in failed pulls and pushes randomly to the registry:

2021-07-03T11:35:58Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}

When I check ingress-nginx logs the authorization header is empty sometimes, so I suspect when the above call fails it gets no token to put in the auth headers leading to the UNAUTHORIZED header error. Question is why this is not always and only sometimes.

[Morriz]

we have the same issue with harbor v2.3.0-047b122c ;(

[Morriz]

I tested on 2.1.5, 2.2.2 and 2.3.0...all suffer this

How are we supposed to pull images again? Isn't this a core part of Harbor?

[Morriz]

I tested with an old setup of ours and this issue is not seen, so excuse me for not trying more exhaustively before wining...for now all we can see is the fact that we upgraded istio to 1.10. Will investigate further...

[sharkymcdongles]

I wonder if maybe there is some issue with the assigning the auth header when a reverse proxy is involved like the nginx ingress.

On Mon, 5 Jul 2021, 21:12 Maurice Faber, @.***> wrote:

I tested with an old setup of ours and this issue is not seen, so excuse me for not trying more exhaustively before wining...for now all we can see is the fact that we upgraded istio to 1.10. Will investigate further...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/15253#issuecomment-874289334, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6KEUMWMCFQMN5OOJ6SMP3TWH73HANCNFSM47W4E4LQ .

[Morriz]

tnx for helping out...I was wondering the same thing. The auth header is set by keycloak and all other internal services that act as client to it have no issues. But it must be related to the header somehow

[Morriz]

I now see one cluster that has no issues with exactly the same setup as on a duplicate cluster I created. One is allowing the login now fine, the other isn't. No pods failing, can't pin down why this intermittent issue exists.

[Morriz]

so at first I thought the logs were relevant, but the working setup logs exactly the same lines, even though auth works:

core 2021-07-05T20:15:40Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="de6b09e5-ce00-4369-a4d6-4a08054a72ce"]: failed to verify secret, username: robot$xxxxxx, error: failed to get oidc user info, error: <QuerySeter> no row found                                                                                            │
core 2021-07-05T20:15:40Z [ERROR] [/pkg/token/token.go:66]: parse token error, token contains an invalid number of segments

The cluster that is failing auth shows nothing different:

core 2021-07-05T20:19:07Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="4a41369c-21e6-4779-927a-45025a9d90c8"]: failed to verify secret, username: robot$team-otomi+kubernetes, error: failed to get oidc user info, error: <QuerySeter> no row found                                                                                                  │
core 2021-07-05T20:19:07Z [ERROR] [/pkg/token/token.go:66]: parse token error, token contains an invalid number of segments

The client that fails:

Error response from daemon: login attempt to https://harbor.xxxxxxx/v2/ failed with status: 401 Unauthorized

[sharkymcdongles]

Ya this is what is most frustrating. It works most of the time but then randomly fails with the header issue.

On Mon, 5 Jul 2021, 22:14 Maurice Faber, @.***> wrote:

I now see one cluster that has no issues with exactly the same setup as on a duplicate cluster I created. One is allowing the login now fine, the other isn't. No pods failing, can't pin down why this intermittent issue exists.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/15253#issuecomment-874311288, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG6KEUNEZZXRE66ZMMDABRLTWIHDZANCNFSM47W4E4LQ .

[Morriz]

Let me also diff the logs in the registry for completeness:

The failing registry:

time="2021-07-05T20:19:08.125619711Z" level=error msg="error authenticating user "admin": authentication failure" go.version=go1.15.12 http.request.host=harbor.xxxxx http.request.id=d02c594b-1180-41fb-8dd1-ee633a1fd7a3 http.request.method=GET http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/" http.request.useragent="docker/20.10.6 go/go1 │
time="2021-07-05T20:19:08.125697914Z" level=warning msg="error authorizing context: basic authentication challenge for realm "harbor-registry-basic-realm": authentication failure" go.version=go1.15.12 http.request.host=harbor.xxxxx http.request.id=d02c594b-1180-41fb-8dd1-ee633a1fd7a3 http.request.method=GET http.request.remoteaddr=127.0.0.1 htt │

The working registry:

127.0.0.6 - - [05/Jul/2021:20:24:36 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/20.10.6 go/go1.13.15 git-commit/8728dd2 kernel/5.10.25-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.6 \\(darwin\\))"

[Morriz]

Another observation:

On the cluster where I am able to successfully docker login with the token, a pullSecret created with the same information is NOT working.

[sharkymcdongles]

So I did some more testing and investigation. It seems the auth header issues and errors were coming from the ingress nginx. Ingress nginx doesn't allow _ in a header by default. After enabling:

enable-underscores-in-headers: 'true'

I was able to run a while loop pulling and removing an image with no failures. That being said I still see NOT_FOUND errors in the logs constantly despite the image existing. I will dig deeper on this issue as it may be unrelated.

examples:

2021-07-06T10:35:03Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:4e006334a6fdea37622f72b21eb75fe1484fc4f20ce8b8526187d6f7bd90a6fe not found"}]}
2021-07-06T10:35:03Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:cdc9dae211b46c95cfcf70f8c98aa6ad0e114439cb5ce2f229b078338f0a4f64 not found"}]}
2021-07-06T10:35:03Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:fe6a4fdbedc0cbc560437fa700b3b034114e31a264f0818d0d32ed2ee6cbe7a3 not found"}]}
2021-07-06T10:35:03Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:e4d0e810d54a9c47e3dface412b0b88045d156b18809a2e0bbfb0fc8a45d8127 not found"}]}
2021-07-06T10:35:03Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:4b4c002ee6ca02473de9ed967cc372717121a986f21ea876f9d854284481a6d0 not found"}]}
2021-07-06T10:35:03Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:7095af798ace32173839a61cbf101048434e1065185c0f29cc888e67158d990b not found"}]}

[sharkymcdongles]

Seems I still see these after the adjustment made above:

021-07-06T12:22:07Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="9d5bf08d0598b184a4a9f7ce39f65f0c"]: failed to verify secret, username: robot$gitlab, error: failed to get oidc user info, error: <QuerySeter> no row found

also still seeing these:

2021-07-06T12:22:07Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"authorize header needed to send HEAD to repository: authorize header needed to send HEAD to repository"}]}

[Morriz]

We have been spending quite some time on figuring this out. The last two lines you mentioned seem to just be noise because of the software design flaw ("trying" a resolution path, expecting that to fail, then going to the next strategy). I wish software makers would stray from creating noise when they could easily avoid it.

[bsinggh]

Hello, did anyone find any solution for this issue?

I just upgraded from 2.2.1 to 2.3.1 and see same error. I am not able to login to docker using robot account.

Jul 26 16:22:22 core[2519]: 2021-07-26T16:22:22Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="092sfa74-c11b-4966-92a9-e6alleda2eae"]: failed to verify secret, username: robot$testupgrade, error: failed to get oidc user info, error: no row found Jul 26 16:22:22 core[2519]: 2021-07-26T16:22:22Z [ERROR] [/pkg/token/token.go:66]: parse token error, token contains an invalid number of segments Jul 26 16:22:22 core[2519]: 2021-07-26T16:22:22Z [ERROR] [/server/middleware/security/robot2.go:47][requestID="092sfa74-c11b-4966-92a9-e6alleda2eae"]: failed to authenticate robot account: robot$testupgrade Jul 26 16:22:22 core[2519]: 2021-07-26T16:22:22Z [ERROR] [/core/auth/authenticator.go:264]: Failed to get user from DB, username: robot$testupgrade, error: user robot$testupgrade not found

[whereismyjetpack]

For me, the upgrade from 2.2 to 2.3 changed the way in which the registry gets it's htpasswd. i was able to remedy this by copying the htpasswd field out of the -harbor-registry secret and placing the same value into the -harbor-registry-htpasswd secret.

[sharkymcdongles]

For me, the upgrade from 2.2 to 2.3 changed the way in which the registry gets it's htpasswd. i was able to remedy this by copying the htpasswd field out of the -harbor-registry secret and placing the same value into the -harbor-registry-htpasswd secret.

For my side it seems this is set correctly, but I suspect perhaps the htpasswd implementation doesn't work with OIDC properly, and maybe switching to token based might work. Will experiment and get back here. NM seems you can no longer use token auth in 2.3+

[sharkymcdongles]

So more digging this error could be solved by adding a check to this method to return nil if the username prefix is the same as the robot prefix:

2021-08-24T11:15:42Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="f7b12e7636f6e9fd320dd6e4ac518d77"]: failed to verify secret, username: ROBOTACCOUNT, error: failed to get oidc user info, error: <QuerySeter> no row found

This would eliminate one of the big error logs we see.

[DennisGlindhart]

This issue seems gone for me with 2.3.2 - Maybe issue https://github.com/goharbor/harbor/issues/15290 had something to do with it?

Anyone else can confirm?

[acobaugh]

Still seeing this with 2.3.3, at least when doing a helm chart push using a robot account.

harbor-test-core-fd897d4f8-z9p8s core 2021-09-28T21:36:49Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="eb2db930c17222d5b43cb97637254b1d"]: failed to verify secret, username: robot+helm+atc135, error: failed to get oidc user info, error: <QuerySeter> no row found
harbor-test-core-fd897d4f8-z9p8s core 2021-09-28T21:36:49Z [INFO] [/server/middleware/security/robot2.go:74][requestID="eb2db930c17222d5b43cb97637254b1d"]: a robot2 security context generated for request GET /service/token

The client then shows:

Error: unexpected response: 401 Unauthorized

[haiwu]

seeing the same for 2.3.3 offline-standalone installation, with oidc enabled, none of robot users would work.

[haiwu]

docker login failure issue (with oidc enabled) seems to be resolved for 2.4 release!

[Morriz]

docker login failure issue (with oidc enabled) seems to be resolved for 2.4 release!

not working for us ;(

On v2.4.0 We see the following output after trying to do a docker login -u 'robot$admin -p '<token>' $server using a newly created robot token with all perms:

│ 2021-10-28T22:45:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d64860f1-d384-4fd3-974d-92659deee639 to the logger for the request GET /v2/                                                                                             │
│ 2021-10-28T22:45:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/                                                                                                                                 │
│ 2021-10-28T22:45:20Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="d64860f1-d384-4fd3-974d-92659deee639"]: an unauthorized security context generated for request GET /v2/                                                           │
│ 2021-10-28T22:45:20Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}                                                                                                                          │
│ 2021-10-28T22:45:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 2520006d-3bc6-4dd5-9b92-582bd615310c to the logger for the request GET /service/token                                                                                   │
│ 2021-10-28T22:45:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /service/token?account=robot%24admin&client_id=docker&offline_token=true&service=harbor-registry                                     │
│ 2021-10-28T22:45:20Z [DEBUG] [/pkg/oidc/secret.go:75]: Verifying the secret for user: robot$admin                                                                                                                                                          │
│ 2021-10-28T22:45:20Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="2520006d-3bc6-4dd5-9b92-582bd615310c"]: failed to verify secret, username: robot$admin, error: failed to get oidc user info, error: <QuerySeter> no row found         │
│ 2021-10-28T22:45:20Z [INFO] [/server/middleware/security/robot.go:71][requestID="2520006d-3bc6-4dd5-9b92-582bd615310c"]: a robot security context generated for request GET /service/token                                                                 │
│ 2021-10-28T22:45:20Z [DEBUG] [/core/service/token/token.go:36]: URL for token request: /service/token?account=robot%24admin&client_id=docker&offline_token=true&service=harbor-registry                                                                    │
│ 2021-10-28T22:45:20Z [DEBUG] [/core/service/token/creator.go:230]: scopes: []                                                                                                                                                                              │
│ 2021-10-28T22:45:20Z [DEBUG] [/core/service/token/authutils.go:50]: scopes: []                                                                                                                                                                             │
│ 2021-10-28T22:45:20Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d9997aa4-fc25-4fba-8291-b5d1718e8fee to the logger for the request GET /v2/                                                                                             │
│ 2021-10-28T22:45:20Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /v2/                                                                                                                                 │
│ 2021-10-28T22:45:20Z [DEBUG] [/pkg/config/manager.go:139]: failed to get key oidc_user_claim, error: the configure value is not set                                                                                                                        │

[Morriz]

Using the same external IDP., on an AWS cluster we see no problems, but on AKS we do. Probably a coincidence.

But, more interesting, after disabling OIDC we see the same errors about the token:

│ 2021-10-29T12:58:00Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 7fc5a5be-f616-4ccd-b926-ffb17c1954bb to the logger for the request GET /v2/                                                                                             │
│ 2021-10-29T12:58:00Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /v2/                                                                                                                                 │
│ 2021-10-29T12:58:00Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="7fc5a5be-f616-4ccd-b926-ffb17c1954bb"]: an unauthorized security context generated for request GET /v2/                                                           │
│ 2021-10-29T12:58:00Z [DEBUG] [/lib/http/error.go:59]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}                                                                                                                          │
│ 2021-10-29T12:58:00Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id f13ad0ae-e103-4869-a54f-a525ada780be to the logger for the request GET /service/token                                                                                   │
│ 2021-10-29T12:58:00Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:52]: In artifact info middleware, url: /service/token?account=robot%24test&client_id=docker&offline_token=true&service=harbor-registry                                      │
│ 2021-10-29T12:58:00Z [ERROR] [/pkg/token/token.go:66]: parse token error, token contains an invalid number of segments                                                                                                                                     │
│ 2021-10-29T12:58:00Z [DEBUG] [/server/middleware/security/robot.go:49][requestID="f13ad0ae-e103-4869-a54f-a525ada780be"]: failed to decrypt robot token of v1 robot: robot$test, as: token contains an invalid number of segments                          │
│ 2021-10-29T12:58:00Z [INFO] [/server/middleware/security/robot2.go:74][requestID="f13ad0ae-e103-4869-a54f-a525ada780be"]: a robot2 security context generated for request GET /service/token                                                               │
│ 2021-10-29T12:58:00Z [DEBUG] [/core/service/token/token.go:36]: URL for token request: /service/token?account=robot%24test&client_id=docker&offline_token=true&service=harbor-registry                                                                     │
│ 2021-10-29T12:58:00Z [DEBUG] [/core/service/token/creator.go:230]: scopes: []                                                                                                                                                                              │
│ 2021-10-29T12:58:00Z [DEBUG] [/core/service/token/authutils.go:50]: scopes: []

[bsinggh]

Did anyone resolve this? I am seeing below for 2.3.4 harbor version.

Nov 12 18:25:54 core[2641]: 2021-11-12T18:25:54Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="07a74387-4cc5-4d94-bde9-b3dge0509939"]: failed to verify secret, username: robot$testupgrade, error: failed to get oidc user info, error: no row found

Nov 12 18:25:54 core[2641]: 2021-11-12T18:25:54Z [ERROR] [/pkg/token/token.go:66]: parse token error, token contains an invalid number of segments

Nov 12 18:25:54 core[2641]: 2021-11-12T18:25:54Z [INFO] [/server/middleware/security/robot2.go:74][requestID="07a74387-4cc5-4d94-bde9-b3dge0509939"]: a robot2 security context generated for request GET /service/token

[j-zimnowoda]

:(

[yogeek]

Same error with 2.4 harbor version (harbor installed with helm, oidc with dex)

[adberger]

Not working for us either with Version v2.4.0-d4affc2e

Robot Account Name: helm-push Robot Account Name (after creation): robot$project+helm-push

2021-12-27T16:28:19Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="f2373e525822377db230e533d50287eb"]: failed to verify secret, username: robot+helm-push, error: failed to get oidc user info, error: <QuerySeter> no row found 2021-12-27T16:28:19Z [ERROR] [/server/middleware/security/basic_auth.go:40][requestID="f2373e525822377db230e533d50287eb"]: failed to authenticate robot+helm-push: not supported

seems to cut out $project

with debug on:

2021-12-27T17:00:10Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6e2afc2e3a79faa224249388a4e82a99 to the logger for the request POST /api/chartrepo/project/charts
2021-12-27T17:00:10Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:53]: In artifact info middleware, url: /api/chartrepo/project/charts
2021-12-27T17:00:10Z [DEBUG] [/pkg/oidc/secret.go:75]: Verifying the secret for user: robot+helm-push
2021-12-27T17:00:10Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="6e2afc2e3a79faa224249388a4e82a99"]: failed to verify secret, username: robot+helm-push, error: failed to get oidc user info, error: <QuerySeter> no row found
2021-12-27T17:00:10Z [DEBUG] [/core/auth/authenticator.go:263]: Failed to get user from DB, username: robot+helm-push, error: user robot+helm-push not found
2021-12-27T17:00:10Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is oidc_auth
2021-12-27T17:00:10Z [ERROR] [/server/middleware/security/basic_auth.go:40][requestID="6e2afc2e3a79faa224249388a4e82a99"]: failed to authenticate robot+helm-push: not supported
2021-12-27T17:00:10Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="6e2afc2e3a79faa224249388a4e82a99"]: an unauthorized security context generated for request POST /api/chartrepo/project/charts

Update:

I got it sorted out, my problem was with helm cm-push Fix 1: Change Robot Account Prefix to something different than $ Fix 2: Escape $ with \ helm cm-push --username=robot\$project+helm-push --password=redacted test-0.1.0.tgz project

[Quantum-Sicarius]

Same issue: Version v2.4.1-c4b06d79

Dec 31 07:58:03 172.19.0.1 core[2343]: 2021-12-31T07:58:03Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="0a5dd53c-531c-47e9-a446-ea3af701d903"]: failed to verify secret, username: robot$up-node, error: failed to get oidc user info, error: <QuerySeter> no row found
Dec 31 07:58:03 172.19.0.1 core[2343]: 2021-12-31T07:58:03Z [ERROR] [/server/middleware/security/basic_auth.go:40][requestID="0a5dd53c-531c-47e9-a446-ea3af701d903"]: failed to authenticate robot$up-node: not supported

[Quantum-Sicarius]

After further investigation, it would appear that Legacy robot accounts appear to be the actual issue as a new robot account works with OIDC.

Version: Version v2.4.1-c4b06d79

[adriannieto-attechnest]

We upgraded from 1.8 to latest version and we are in the same situation. This happens in automated robot accounts like trivy one too.

`harbor-core-68f446c9bf-pjm8v core 2022-02-04T06:55:09Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="1a3d6e25-9104-44f7-99a7-1dc56a507c62"]: failed to verify secret, username: robot$xxx+Trivy-cbdab39d-8585-11ec-aabe-765eaaf3b54e, error: failed to get oidc user info, error: <QuerySeter> no row found

[j14s]

My logs are getting absolutely flooded right now with OIDC errors like this.

[sremp]

We've been receiving these same error messages for some time now, however our GitHub Actions image publishing was working up until last Thursday when our Harbor Helm release auto-updated. Now, all we are seeing is: [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="xxxxxxxx"]: failed to verify secret, username: xxxx, error: failed to get oidc user info, error: <QuerySeter> no row

[warmfusion]

After further investigation, it would appear that Legacy robot accounts appear to be the actual issue as a new robot account works with OIDC.

Version: Version v2.4.1-c4b06d79

We're also using Version v2.4.1-c4b06d79 but a robot account created on the 31st March (6 days ago) shows the same error.

We're using an @ rather than $ for the robot separator, have been upgrading harbor since the 1.x era so there may be something with an upgrade process causing issues but its not clear.

[eg-ops]

In our case "Garbage Collection" in the settings tab fixed the problem. The main problem seems to be in running out of memory in minio service.

[zdhamasha]

any updates regarding to this issue ?

[vwatinteg]

Same as zdhamasha, any updates? We're on CHART NAME: harbor CHART VERSION: 11.1.0 APP VERSION: 2.4.0

Been getting these logs for months...

[IDerr]

I'm getting these logs too :(

[naitmare01]

Same for me.

[nedhanks]

I'm getting these errors on a fresh install of 2.5.3. OIDC is configured, and seems to be work ok. Created a robot account to do replication from another harbor install and seeing a lot of these errors.

NAME    NAMESPACE       REVISION    UPDATED                                 STATUS      CHART           APP VERSION
marina  harbor-marina   1           2022-07-29 19:28:33.161549 -0600 MDT    deployed    harbor-1.9.3    2.5.3

Also, I'm unable to login using docker, but I can login to the UI. I copied the token from my profile.

docker login marina.[redacted] -u ned_hanks -p [redacted]
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://marina.octanner.io/v2/": unauthorized: authentication required

2022-08-12T23:27:03Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="03bc7f14-0fa2-4eda-9c16-eb1d9877043a"]: failed to verify secret, username: ned_hanks, error: failed to get oidc user info, error: <QuerySeter> no row found
2022-08-12T23:27:03Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.2.48.0, 10.2.9.212" requestID="03bc7f14-0fa2-4eda-9c16-eb1d9877043a" user agent="docker/20.10.12 go/go1.16.12 git-commit/459d0df kernel/5.10.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.12 \(darwin\))"]: failed to authenticate user:ned_hanks, error:not supported

[adriannieto-attechnest]

@wy65701436 could you delete Ned comment? He is leaking credentials by mistake

[danielfm]

I'm experiencing this issue too.

Using a robot account in helm repo add command fails, but the same account works when using helm registry login.

$ helm registry login <url> --username 'robot$<user>' --password '<token>'
WARNING: Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

$ helm repo add test <url>/chartrepo --username 'robot$<user>' --password '<token>'
Error: looks like "<url>/chartrepo" is not a valid chart repository or cannot be reached: failed to fetch <url>/chartrepo/index.yaml : 403 Forbidden

I can use a robot account in docker login as well, so maybe that issue is specific to the chartmuseum integration?

Everything works as expected when using a user token (both the admin user and users onboarded via OIDC).

I'm running v2.6.0, fresh install.

---

Update 2022-09-09: Apparently trying to add a Helm repository using the <harbor-url>/chartrepo URL does not work, but using the <harbor-url>/chartrepo/<project-name> URL does work:

$ helm repo add test <url>/chartrepo --username 'robot$<user>' --password '<token>'
Error: looks like "<url>/chartrepo" is not a valid chart repository or cannot be reached: failed to fetch <url>/chartrepo/index.yaml : 403 Forbidden

$ helm repo add test <url>/chartrepo/<project> --username 'robot$<user>' --password '<token>'
"test" has been added to your repositories

That goes against the information available in the documentation:

Add Harbor as a unified single index entry point

With this mode Helm can be made aware of all the charts located in different projects and which are accessible by the currently authenticated user.

helm repo add --ca-file ca.crt --username=admin --password=Passw0rd myrepo https://xx.xx.xx.xx/chartrepo

If I log in using a non-robot account, authenticating to /chartrepo works.

Helm version: 3.9.4

[jimsnab]

I hit this yesterday on a fresh install of Harbor via v1.10.0 Helm chart. I have Jenkins build automation that uploads build results to Harbor, and builds were suddenly failing with 502 Bad Gateway. I got this no row found error and I searched all day yesterday to try and figure out why. Resumed investigating today, then inexplicably a moment ago the push errored but resulted in success response anyway. I did nothing but re-run the build in attempts to find more details in logs.

2022-09-09T18:44:38Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="..."]: failed to verify secret, username: robot$my-environment+jenkins, error: failed to get oidc user info, error: <QuerySeter> no row found
2022-09-09T18:44:38Z [INFO] [/server/middleware/security/robot.go:71][requestID="..."]: a robot security context generated for request GET /service/token
2022-09-09T18:44:39Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="..."]: failed to verify secret, username: robot$my-environment+jenkins, error: failed to get oidc user info, error: <QuerySeter> no row found
2022-09-09T18:44:39Z [INFO] [/server/middleware/security/robot.go:71][requestID="..."]: a robot security context generated for request GET /service/token
2022-09-09T18:46:15Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="..."]: failed to verify secret, username: robot$my-environment+jenkins, error: failed to get oidc user info, error: <QuerySeter> no row found
2022-09-09T18:46:15Z [INFO] [/server/middleware/security/robot.go:71][requestID="..."]: a robot security context generated for request GET /service/token
2022-09-09T18:46:29Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PUSH_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[34d6a...] Digest-sha256:575d18... Operator-robot$my-environment+jenkins OccurAt-2022-09-09 18:46:29
2022-09-09T18:46:29Z [INFO] [/controller/event/handler/webhook/artifact/artifact.go:75]: []
2022-09-09T18:46:29Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'PUSH_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[34d6a...] Digest-sha256:575d18... Operator-robot$my-environment+jenkins OccurAt-2022-09-09 18:46:29
2022-09-09T18:46:29Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'PUSH_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[34d6a...] Digest-sha256:575d18... Operator-robot$my-environment+jenkins OccurAt-2022-09-09 18:46:29
2022-09-09T18:46:29Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'Replication' on topic 'PUSH_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[34d6a...] Digest-sha256:575d18... Operator-robot$my-environment+jenkins OccurAt-2022-09-09 18:46:29
2022-09-09T18:46:30Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'P2PPreheat' on topic 'PUSH_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[34d6a...] Digest-sha256:575d18... Operator-robot$my-environment+jenkins OccurAt-2022-09-09 18:46:29
2022-09-09T18:46:43Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="..."]: failed to verify secret, username: robot$my-environment+Trivy-b830189f-306f-11ed-8235-06a8e23d6bf2, error: failed to get oidc user info, error: <QuerySeter> no row found
2022-09-09T18:46:43Z [INFO] [/server/middleware/security/robot.go:71][requestID="..."]: a robot security context generated for request GET /service/token
2022-09-09T18:46:43Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PULL_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[] Digest-sha256:575d18... Operator-robot$my-environment+Trivy-b830189f-306f-11ed-8235-06a8e23d6bf2 OccurAt-2022-09-09 18:46:43
2022-09-09T18:46:43Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'PULL_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[] Digest-sha256:575d18... Operator-robot$my-environment+Trivy-b830189f-306f-11ed-8235-06a8e23d6bf2 OccurAt-2022-09-09 18:46:43
2022-09-09T18:46:43Z [INFO] [/controller/event/handler/webhook/artifact/artifact.go:75]: []
2022-09-09T18:46:43Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'PULL_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[] Digest-sha256:575d18... Operator-robot$my-environment+Trivy-b830189f-306f-11ed-8235-06a8e23d6bf2 OccurAt-2022-09-09 18:46:43
2022-09-09T18:47:09Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PULL_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[] Digest-sha256:575d18... Operator- OccurAt-2022-09-09 18:47:09
2022-09-09T18:47:09Z [INFO] [/controller/event/handler/webhook/artifact/artifact.go:75]: []
2022-09-09T18:47:09Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'PULL_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[] Digest-sha256:575d18... Operator- OccurAt-2022-09-09 18:47:09
2022-09-09T18:47:09Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'PULL_ARTIFACT': ID-1, Repository-my-environment/example-project Tags-[] Digest-sha256:575d18... Operator- OccurAt-2022-09-09 18:47:09
2022-09-09T18:47:18Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ScanWebhook' on topic 'SCANNING_COMPLETED': Artifact-&{NamespaceID:2 Repository:my-environment/example-project Tag:34d6a2... Digest:sha256:575d18... MimeType:application/vnd.docker.distribution.manifest.v2+json} Operator-auto OccurAt-2022-09-09 18:47:18
2022-09-09T18:47:19Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'P2PPreheat' on topic 'SCANNING_COMPLETED': Artifact-&{NamespaceID:2 Repository:my-environment/example-project Tag:34d6a2... Digest:sha256:575d18... MimeType:application/vnd.docker.distribution.manifest.v2+json} Operator-auto OccurAt-2022-09-09 18:47:18

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[cesarb1392]

I'm still running into this issue. Deploying Harbor using the Helm chart version v1.10.2

[ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="552f33a7-4a03-4b04-8782-f5aa6152b578"]: failed to verify secret, username: robot$account, error: failed to get oidc user info, error: <QuerySeter> no row found
[INFO] [/server/middleware/security/robot.go:71][requestID="552f33a7-4a03-4b04-8782-f5aa6152b578"]: a robot security context generated for request GET /service/token

[steadyk]

I also see this on v1.10.2, but with slightly different errors:

2022-11-21T07:56:55Z [ERROR] [/server/middleware/security/basic_auth.go:72][client IP="10.244.0.70:55126" requestID="4251fe16-8e20-4c74-ad08-774836c87951" user agent="curl/7.72.0-DEV"]: failed to authenticate user:admin, error:Failed to authenticate user, due to error 'Invalid credentials'
2022-11-21T07:56:57Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="70a43bf6-3933-4c4a-b79d-9b93ab2a4485"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2022-11-21T07:56:57Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="3eb4527f-1039-463f-ba57-4d2f10a5122c"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2022-11-21T07:56:57Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="72f15e3f-8f9d-45c6-8e5e-e257e4e8f216"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found

Mabye that also happens due to migration from v.1.10.1 -> v1.10.2. Could be also an issue on our local environment.

[rgarcia89]

Any update here? I am also see the core pod logs getting filled with that