dioguerra commented 1 year ago

I upgraded a cluster from 2.5.4 to 2.7.1. Trying to export vulnerability report (also generated from the previous version - not sure that matters) I'm having an error on the UI.

Looks like this has something to do with making the file available to download. No problem on a repo with no vuln.

Expected behavior and actual behavior: Be able to download the report, or no error

Steps to reproduce the problem: Follow https://goharbor.io/blog/harbor-2.6/ on CVE export

Versions:

harbor version: 2.7.1
harbor-helm version: 1.11.1

Debug

Usefull to knows:

Using OIDC
S3 backend remote

Jobservice logs show:

 2023-04-04T16:46:48Z [DEBUG] [/jobservice/api/handler.go:281]: Serve http request 'POST /api/v1/jobs': 202 {"job":{"id":"4208f90039561aeb14b8a68e","status":"Pending","name":"SCAN_DATA_EXPORT","kind":"Generic","unique":false,"ref_link":"/api/v1/jobs/4208f90039561aeb14b8a68e","enqueue_time":1680626808,"update_time":1680626808,"web_hook_url":"http://kops-registry-staging-harbor-core:80/service/notifications/tasks/14","parameters":{"JobId":14,"Request":{"CVEIds":"","JobName":"","Labels":[],"Projects":[2],"Repositories":"","Tags":"","UserID":3,"UserName":"dtomasgu"},"mode":"export"}}}
2023-04-04T16:46:49Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"SCAN_DATA_EXPORT","id":"4208f90039561aeb14b8a68e","t":1680626808,"args":null}
2023-04-04T16:46:49Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://kops-registry-staging-harbor-core:80/api/v2.0/internalconfig
2023-04-04T16:46:49Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://kops-registry-staging-harbor-core:80/api/v2.0/internalconfig
2023-04-04T16:46:49Z [DEBUG] [/jobservice/hook/hook_agent.go:118]: Hook event is successfully sent: status change: job=4208f90039561aeb14b8a68e, status=Running, revision=1680626808->http://kops-registry-staging-harbor-core:80/service/notifications/tasks/14
2023-04-04T16:46:49Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:79]: Scan data export job started in mode : export
2023-04-04T16:46:49Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:175]: Created CSV export file /tmp/scandata_export_14.csv
2023-04-04T16:46:49Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:181]: Request for export : map[CVEIds: JobName: Labels:[] Projects:[2] Repositories: Tags: UserID:3 UserName:dtomasgu]
2023-04-04T16:46:49Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:258]: Export Group Id = 0, Job Id = 14, Page Number = 1, Page Size = 100000 Num Records = 139
2023-04-04T16:46:49Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:96]: Export Job Id = 14, FileName = /tmp/scandata_export_14.csv, Hash = sha256:7a3a514493195cb801b0c128567355d02e95d94eb71be7edcaa55303722ca082
2023-04-04T16:46:49Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:106]: Creating repository for CSV file with blob : scandata_export_14
2023-04-04T16:46:49Z [INFO] [/jobservice/job/impl/scandataexport/scan_data_export.go:112]: Export Job Id = 14. CSV file size: 53263
2023-04-04T16:46:49Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: http://kops-registry-staging-harbor-core:80/api/v2.0/internalconfig
2023-04-04T16:46:49Z [ERROR] [/pkg/systemartifact/manager.go:111]: Error creating system artifact record for scan_data_export/scandata_export_14/sha256:7a3a514493195cb801b0c128567355d02e95d94eb71be7edcaa55303722ca082: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"sys_harb0r/scan_data_export/scandata_export_14","Action":"pull"},{"Type":"repository","Class":"","Name":"sys_harb0r/scan_data_export/scandata_export_14","Action":"push"}]}]}
2023-04-04T16:46:49Z [ERROR] [/jobservice/job/impl/scandataexport/scan_data_export.go:130]: Export Job Id = 14. Error when persisting report file /tmp/scandata_export_14.csv to persistent storage: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"sys_harb0r/scan_data_export/scandata_export_14","Action":"pull"},{"Type":"repository","Class":"","Name":"sys_harb0r/scan_data_export/scandata_export_14","Action":"push"}]}]}
2023-04-04T16:46:49Z [ERROR] [/jobservice/runner/redis.go:123]: Job 'SCAN_DATA_EXPORT:4208f90039561aeb14b8a68e' exit with error: run error: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"sys_harb0r/scan_data_export/scandata_export_14","Action":"pull"},{"Type":"repository","Class":"","Name":"sys_harb0r/scan_data_export/scandata_export_14","Action":"push"}]}]}
2023-04-04T16:46:49Z [DEBUG] [/jobservice/hook/hook_agent.go:118]: Hook event is successfully sent: status change: job=4208f90039561aeb14b8a68e, status=Error, revision=1680626808->http://kops-registry-staging-harbor-core:80/service/notifications/tasks/14

Core (not much use)

2023-04-04T17:29:02Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 40a600625f4ebc154def63779ebb908e to the logger for the request GET /api/v2.0/export/cve/executions
2023-04-04T17:29:02Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/export/cve/executions
2023-04-04T17:29:02Z [DEBUG] [/server/middleware/security/session.go:47][requestID="40a600625f4ebc154def63779ebb908e"]: a session security context generated for request GET /api/v2.0/export/cve/executions
2023-04-04T17:29:02Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /project/2/export-cve
2023-04-04T17:29:02Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /project/3/export-cve
2023-04-04T17:29:03Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require create action for resource /project/2/export-cve
2023-04-04T17:29:03Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /project/2/export-cve
2023-04-04T17:29:03Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require list action for resource /project/2/export-cve
2023-04-04T17:29:06Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 944e0e1e8140a617c0e5d7544dc2cef4 to the logger for the request POST /api/v2.0/export/cve
2023-04-04T17:29:06Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/export/cve
2023-04-04T17:29:06Z [DEBUG] [/server/middleware/security/session.go:47][requestID="944e0e1e8140a617c0e5d7544dc2cef4"]: a session security context generated for request POST /api/v2.0/export/cve
2023-04-04T17:29:06Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require create action for resource /project/2/export-cve
2023-04-04T17:29:06Z [INFO] [/controller/scandataexport/execution.go:123][requestID="944e0e1e8140a617c0e5d7544dc2cef4"]: Created an execution record with id : 23 for vendorID: 3
2023-04-04T17:29:06Z [INFO] [/controller/scandataexport/execution.go:151][requestID="944e0e1e8140a617c0e5d7544dc2cef4"]: Created job for scan data export successfully
2023-04-04T17:29:06Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 8d7cbadb93a3eaf04e6b9b06135c9b5c to the logger for the request GET /api/v2.0/export/cve/executions
2023-04-04T17:29:06Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/export/cve/executions
2023-04-04T17:29:06Z [DEBUG] [/server/middleware/security/session.go:47][requestID="8d7cbadb93a3eaf04e6b9b06135c9b5c"]: a session security context generated for request GET /api/v2.0/export/cve/executions
2023-04-04T17:29:06Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /project/2/export-cve
2023-04-04T17:29:06Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /project/3/export-cve
2023-04-04T17:29:13Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id a277dd2242691a738dab52125d604da2 to the logger for the request GET /api/v2.0/export/cve/executions
2023-04-04T17:29:13Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/export/cve/executions
2023-04-04T17:29:13Z [DEBUG] [/server/middleware/security/session.go:47][requestID="a277dd2242691a738dab52125d604da2"]: a session security context generated for request GET /api/v2.0/export/cve/executions
2023-04-04T17:29:13Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /project/2/export-cve
2023-04-04T17:29:13Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /project/3/export-cve

Curious permissions is asking to access both project 2 and 3, when this time, i only asked to get report for project 2 (and multiple projects seems to not be available at the time)

Also did a quick pick search on some src code (i think not relevant tho) and found a similar error related to Audit Log Forward (need to check again when i have time)?

dioguerra commented 1 year ago

Uhm, I just gave a try to do image scanning. And I still have symptoms from a problem im still investigating with trivy access to manifests. please hold until I have more data.

Is this related to the CVE recently announced?

zyyw commented 1 year ago

Is this related to the CVE recently announced?

Which CVE are you referring to? Please provide us with more details, thanks.

zyyw commented 1 year ago

might be related to this issue:

https://github.com/goharbor/harbor/issues/18380

which will be fixed in v2.6.5, v2.7.2 and v2.8.0

chlins commented 1 year ago

might be related to this issue:

CVE export error "404 page not found" #18380

which will be fixed in v2.6.5, v2.7.2 and v2.8.0

I think the issue is not related with the fixed one, they have different error code.

chlins commented 1 year ago

@dioguerra Could you push the images successfully in the current harbor?

dioguerra commented 1 year ago

Hello @chlins @zyyw : So this is the status of my problem. Everything that seems to interact with core directly, if it needs authentication it fails. This includes (nerdctl login, trivy, and possibly also CVE export?)

I tried to create a fresh new install (cleaning up PVC's and database) but still does not work. I'm not sure i might have miss configured something. From my diffs this does not seem to be the case tho.

In the core pod, if i try to nerdctl login registry.foo.bar -u me and submit the password, the user seems to be atributed a token, but some validation fails after. There seems to be a new field under System->OIDC->OIDC Group Filter (which btw, prints errors if left blank)

nerdctl login registry-staging.foo.bar -u dtomasgu

2023-04-11T15:40:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 8636715a62225fa9c0653b88fc214b26 to the logger for the request GET /v2/
2023-04-11T15:40:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-04-11T15:40:34Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="8636715a62225fa9c0653b88fc214b26"]: an unauthorized security context generated for request GET /v2/
2023-04-11T15:40:34Z [DEBUG] [/lib/http/error.go:61]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}
2023-04-11T15:40:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0aafce8d274d8d7076f00e54b4fe3726 to the logger for the request POST /service/token
2023-04-11T15:40:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /service/token
2023-04-11T15:40:34Z [DEBUG] [/server/middleware/security/unauthorized.go:28][requestID="0aafce8d274d8d7076f00e54b4fe3726"]: an unauthorized security context generated for request POST /service/token
2023-04-11T15:40:34Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id f69b70268bc2a598a6d45c5b68743281 to the logger for the request GET /service/token
2023-04-11T15:40:34Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /service/token?offline_token=true&service=harbor-registry
2023-04-11T15:40:34Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-04-11T15:40:34Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-04-11T15:40:34Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-04-11T15:40:35Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="f69b70268bc2a598a6d45c5b68743281"]: an OIDC CLI security context generated for request GET /service/token
2023-04-11T15:40:35Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?offline_token=true&service=harbor-registry
2023-04-11T15:40:35Z [DEBUG] [/core/service/token/creator.go:231]: scopes: []
2023-04-11T15:40:35Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: []

From what i see from the token, the groups seem to be there, there is this section which i'm not sure it matters:

"resource_access":{"registry":{"roles":["default-role"]}}

So, to respond to yout question, i cannot push any images due to authentication problems. I can (tho) navigate harbor using the portal, which i find strange

Seems to be related to OIDC somehow?

nerdctl login times out after a while with too many retries which i can see in the core logs

chlins commented 1 year ago

That seems the authorization of harbor is not normal, so could you try to login with admin account instead of OIDC user?

dioguerra commented 1 year ago

Similar thing happens with admin user, error is different tho.

2023-04-12T12:46:25Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/security/basic_auth.go:79][requestID="1fa1f90c7ca63ac32bb0eec331abdfa2"]: a basic auth security context generated for request GET /v2/
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0b201fa03ec6efec828e6ce3e11baaa4 to the logger for the request GET /v2/
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-04-12T12:46:25Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: admin
2023-04-12T12:46:25Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="0b201fa03ec6efec828e6ce3e11baaa4"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found

chlins commented 1 year ago

Similar thing happens with admin user, error is different tho.

2023-04-12T12:46:25Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/security/basic_auth.go:79][requestID="1fa1f90c7ca63ac32bb0eec331abdfa2"]: a basic auth security context generated for request GET /v2/
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0b201fa03ec6efec828e6ce3e11baaa4 to the logger for the request GET /v2/
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-04-12T12:46:25Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: admin
2023-04-12T12:46:25Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="0b201fa03ec6efec828e6ce3e11baaa4"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found

@stonezdj Could you help to take a look?

stonezdj commented 1 year ago

Can you please upload the job log, you could find the job log

docker exec -it harbor-db bash
psql -U postgres -d registry
select * from execution where vendor_type = 'SCAN_DATA_EXPORT' order by start_time desc limit 10
select job_id from task where execution_id = <execution id>

Get the log in /data/job_log/ if you are running harbor in k8s, you need to ssh to the job service container to get the log in /var/log/jobs. for docker-compose, go to the local directory under /data/job_log.

cat .log

chlins commented 1 year ago

Similar thing happens with admin user, error is different tho.

2023-04-12T12:46:25Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/security/basic_auth.go:79][requestID="1fa1f90c7ca63ac32bb0eec331abdfa2"]: a basic auth security context generated for request GET /v2/
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0b201fa03ec6efec828e6ce3e11baaa4 to the logger for the request GET /v2/
2023-04-12T12:46:25Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-04-12T12:46:25Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: admin
2023-04-12T12:46:25Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="0b201fa03ec6efec828e6ce3e11baaa4"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found

@dioguerra You mean that the admin user cannot login to harbor as well?

dioguerra commented 1 year ago

nnot login to harbor as well?

@chlins This seems to be the case. Both Admin/OIDC User cannot interact via docker CLI starting from docker login

@stonezdj From the joblogs, all jobs should be pushed both to database and stdout (as per helm config) I can't do this at this moment as I can't push/pull any artifact to/from the database.

Before, I tried to generate this reports with images (and scan results) that where in the registry before migrating (from 2.5 to 2.7) but i noticed scans where not running in addition to the reports problems.
To try and fix this I did a clean installation:
1. resize to 0 all deployments
2. Drop database and recreate
3. Delete PVC and PV, followed by deleting redis pod
4. Recreate PVC to allow redis to come back up
5. resize all previously scalled down deployments to 1

I have the jobservice stdout logs tho, but dont seem to be able to find the original logs from the first issue comment (even accounting for some sort of time skew). Logs just not there, eitherway, should have no more information that the original post... https://github.com/goharbor/harbor/issues/18480#issue-1654281360

What can I do to better sanitize the environment?

dioguerra commented 1 year ago

Just gave a try with the v2.7.2-rc1 image and still not able to docker login

chlins commented 1 year ago

Could you provide the error message when using the admin account for docker login, as well as the core logs at the same time? It seems your issue is not related with CVE export or job service, the root cause is the login failure.

dioguerra commented 1 year ago

This looks like something in OIDC that changed:

nerdctl login harbor.foo.bar -u dtomasgu --debug-full
Enter Password: 
DEBU[0004] Ignoring hosts dir "/home/dtomasgu/.config/containerd/certs.d"  error="stat /home/dtomasgu/.config/containerd/certs.d: no such file or directory"
DEBU[0004] Ignoring hosts dir "/home/dtomasgu/.config/docker/certs.d"  error="stat /home/dtomasgu/.config/docker/certs.d: no such file or directory"
DEBU[0004] len(regHosts)=1                              
DEBU[0004] no scope specified for token auth challenge   host=harbor.foo.bar
ERRO[0013] failed to call tryLoginWithRegHost            error="too many 401 (probably)" i=0
FATA[0013] too many 401 (probably)

From the core pod, authentication seems to be allowed, but for some reason the core pod cycles through requests:

$ kn harbor logs -f kops-harbor-harbor-core-56ddf564fb-n4f2z | grep -v "oidc_group_filter" | grep -v '/api/v2.0/ping' | grep -v 'an unauthorized security'
2023-05-17T08:58:21Z [DEBUG] [/pkg/config/manager.go:140]: failed to get key ldap_group_attribute_name, error: the configure value is not set, maybe default value not defined before get
2023-05-17T08:58:21Z [DEBUG] [/pkg/config/manager.go:140]: failed to get key audit_log_forward_endpoint, error: the configure value is not set, maybe default value not defined before get
2023-05-17T08:58:21Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0c3fa94b806aa8d75a724365cdf65211 to the logger for the request GET /api/v2.0/configurations
2023-05-17T08:58:21Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/configurations
2023-05-17T08:58:21Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d54cc67bb40e20df2e6d6bc768558f8c to the logger for the request GET /api/v2.0/systeminfo
2023-05-17T08:58:21Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/systeminfo
2023-05-17T08:58:21Z [DEBUG] [/server/middleware/security/session.go:47][requestID="0c3fa94b806aa8d75a724365cdf65211"]: a session security context generated for request GET /api/v2.0/configurations
2023-05-17T08:58:21Z [DEBUG] [/server/middleware/security/session.go:47][requestID="d54cc67bb40e20df2e6d6bc768558f8c"]: a session security context generated for request GET /api/v2.0/systeminfo
2023-05-17T08:58:21Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator dtomasgu require read action for resource /system/configuration
2023-05-17T09:03:30Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 801a7bb613ef8aaac538f2703969ac77 to the logger for the request GET /v2/
2023-05-17T09:03:30Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:30Z [DEBUG] [/lib/http/error.go:61]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}
2023-05-17T09:03:30Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 47697b7febe3123767c84d5d84624747 to the logger for the request POST /service/token
2023-05-17T09:03:30Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /service/token
2023-05-17T09:03:30Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 68e04e450718c5ce0ea8f7a943082b84 to the logger for the request GET /service/token
2023-05-17T09:03:30Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /service/token?offline_token=true&service=harbor-registry
2023-05-17T09:03:30Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:30Z [DEBUG] [/pkg/oidc/secret.go:102]: Refreshing token
2023-05-17T09:03:31Z [DEBUG] [/pkg/oidc/secret.go:118]: Token refreshed and persisted
2023-05-17T09:03:31Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:31Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:32Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="68e04e450718c5ce0ea8f7a943082b84"]: an OIDC CLI security context generated for request GET /service/token
2023-05-17T09:03:32Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?offline_token=true&service=harbor-registry
2023-05-17T09:03:32Z [DEBUG] [/core/service/token/creator.go:231]: scopes: []
2023-05-17T09:03:32Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: []
2023-05-17T09:03:32Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id aa73884e8856d07aac4a5bd7a16e0571 to the logger for the request GET /v2/
2023-05-17T09:03:32Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:32Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 3239cea07a49e637965a24c807a09a15 to the logger for the request GET /v2/
2023-05-17T09:03:32Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:32Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:32Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:32Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:33Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="3239cea07a49e637965a24c807a09a15"]: an OIDC CLI security context generated for request GET /v2/
2023-05-17T09:03:33Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 6f5e1b806234dc17140a84aa5b68328d to the logger for the request GET /v2/
2023-05-17T09:03:33Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:33Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:33Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:33Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:33Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="6f5e1b806234dc17140a84aa5b68328d"]: an OIDC CLI security context generated for request GET /v2/
2023-05-17T09:03:33Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 73214f4169ae376a612bcc7a8e4a3ce3 to the logger for the request GET /v2/
2023-05-17T09:03:33Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:33Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:33Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:34Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:35Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="73214f4169ae376a612bcc7a8e4a3ce3"]: an OIDC CLI security context generated for request GET /v2/
2023-05-17T09:03:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id b86c7350de1d5090d3f50406a4e97ecb to the logger for the request GET /v2/
2023-05-17T09:03:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:35Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:35Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:35Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:35Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="b86c7350de1d5090d3f50406a4e97ecb"]: an OIDC CLI security context generated for request GET /v2/
2023-05-17T09:03:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id bc998e1080af57bcca4f837b046d08c8 to the logger for the request GET /v2/
2023-05-17T09:03:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:35Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:35Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:36Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:36Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="bc998e1080af57bcca4f837b046d08c8"]: an OIDC CLI security context generated for request GET /v2/
2023-05-17T09:03:36Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0e77f30234dd2da73da2a7d783882ba7 to the logger for the request GET /v2/
2023-05-17T09:03:36Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:36Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:36Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:37Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:37Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="0e77f30234dd2da73da2a7d783882ba7"]: an OIDC CLI security context generated for request GET /v2/
2023-05-17T09:03:37Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id f5d760faa0790e680f5c0f013c92274d to the logger for the request GET /v2/
2023-05-17T09:03:37Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:37Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu
2023-05-17T09:03:37Z [DEBUG] [/pkg/oidc/helper.go:210]: Raw ID token for verification: REDACTED
2023-05-17T09:03:38Z [DEBUG] [/pkg/oidc/helper.go:414]: populateGroupsDB, group filter 
2023-05-17T09:03:38Z [DEBUG] [/server/middleware/security/oidc_cli.go:71][requestID="f5d760faa0790e680f5c0f013c92274d"]: an OIDC CLI security context generated for request GET /v2/
2023-05-17T09:03:38Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 0c386e778877743c4ada53fb7833ca52 to the logger for the request GET /v2/
2023-05-17T09:03:38Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:03:38Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: dtomasgu

From the configurations I have OIDC_SCOPE=openid (this works in our current production service) and also tried to add OIDC_SCOPE=openid,offline_access

for the admin user:

nerdctl login harbor.foo.bar -u admin --debug-full
Enter Password: 
DEBU[0003] Ignoring hosts dir "/home/dtomasgu/.config/containerd/certs.d"  error="stat /home/dtomasgu/.config/containerd/certs.d: no such file or directory"
DEBU[0003] Ignoring hosts dir "/home/dtomasgu/.config/docker/certs.d"  error="stat /home/dtomasgu/.config/docker/certs.d: no such file or directory"
DEBU[0003] len(regHosts)=1                              
DEBU[0003] no scope specified for token auth challenge   host=harbor.foo.bar
ERRO[0004] failed to call tryLoginWithRegHost            error="too many 401 (probably)" i=0
FATA[0004] too many 401 (probably)

And for the core container

2023-05-17T09:10:15Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/scanners?page_size=15&page=1
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/security/session.go:47][requestID="2a5252f20089d1dd502a3c41e187a2d6"]: a session security context generated for request GET /api/v2.0/scanners
2023-05-17T09:10:15Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator admin require list action for resource /system/scanner
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id fff702e9e1b541b142ab19ec4fe383ad to the logger for the request GET /api/v2.0/configurations
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/configurations
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id b40a10d4ce62625a2c4a43a0f90f9b1c to the logger for the request GET /api/v2.0/registries
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/registries?q=type%3D%7Bdocker-hub%20harbor%20azure-acr%20aws-ecr%20google-gcr%20quay%20docker-registry%20github-ghcr%20jfrog-artifactory%7D&page_size=100&page=1
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 28d93ec32216f334ca9c26c0a565f36f to the logger for the request GET /api/v2.0/statistics
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/statistics
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/security/session.go:47][requestID="b40a10d4ce62625a2c4a43a0f90f9b1c"]: a session security context generated for request GET /api/v2.0/registries
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/security/session.go:47][requestID="fff702e9e1b541b142ab19ec4fe383ad"]: a session security context generated for request GET /api/v2.0/configurations
2023-05-17T09:10:15Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator admin require read action for resource /system/configuration
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/security/session.go:47][requestID="28d93ec32216f334ca9c26c0a565f36f"]: a session security context generated for request GET /api/v2.0/statistics
2023-05-17T09:10:15Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator admin require list action for resource /system/registry
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 93f3813cffbde5967337207d7a566c75 to the logger for the request GET /api/v2.0/projects
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/projects?page=1&page_size=15
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/security/session.go:47][requestID="93f3813cffbde5967337207d7a566c75"]: a session security context generated for request GET /api/v2.0/projects
2023-05-17T09:10:15Z [DEBUG] [/pkg/permission/evaluator/admin/admin.go:35]: system administrator admin require list action for resource /system/project
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 85b0273f75be8ba0e0927528bd847f4e to the logger for the request GET /api/v2.0/export/cve/executions
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /api/v2.0/export/cve/executions
2023-05-17T09:10:15Z [DEBUG] [/server/middleware/security/session.go:47][requestID="85b0273f75be8ba0e0927528bd847f4e"]: a session security context generated for request GET /api/v2.0/export/cve/executions
2023-05-17T09:10:15Z [DEBUG] [/chartserver/handler_repo.go:166]: Getting index.yaml from 'http://kops-harbor-harbor-chartmuseum/library/index.yaml'
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 448eec76431c6172f1ffb9708ea43b4a to the logger for the request GET /v2/
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:10:35Z [DEBUG] [/lib/http/error.go:61]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id a45c32fd59a2e82c26843d3a2ec69d56 to the logger for the request POST /service/token
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /service/token
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id db75c1743b433ff75de710193201bc2c to the logger for the request GET /service/token
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /service/token?offline_token=true&service=harbor-registry
2023-05-17T09:10:35Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: admin
2023-05-17T09:10:35Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="db75c1743b433ff75de710193201bc2c"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2023-05-17T09:10:35Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/security/basic_auth.go:79][requestID="db75c1743b433ff75de710193201bc2c"]: a basic auth security context generated for request GET /service/token
2023-05-17T09:10:35Z [DEBUG] [/core/service/token/token.go:37]: URL for token request: /service/token?offline_token=true&service=harbor-registry
2023-05-17T09:10:35Z [DEBUG] [/core/service/token/creator.go:231]: scopes: []
2023-05-17T09:10:35Z [DEBUG] [/core/service/token/authutils.go:51]: scopes: []
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id a51929d40f949e8bc9f7a4a880e46017 to the logger for the request GET /v2/
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id d529395967b12c45040e4fc85bd003c4 to the logger for the request GET /v2/
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:10:35Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: admin
2023-05-17T09:10:35Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="d529395967b12c45040e4fc85bd003c4"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2023-05-17T09:10:35Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/security/basic_auth.go:79][requestID="d529395967b12c45040e4fc85bd003c4"]: a basic auth security context generated for request GET /v2/
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/log/log.go:30]: attach request id 8a4a009567f169bb3836ddfacc2bc1ac to the logger for the request GET /v2/
2023-05-17T09:10:35Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:54]: In artifact info middleware, url: /v2/
2023-05-17T09:10:35Z [DEBUG] [/pkg/oidc/secret.go:73]: Verifying the secret for user: admin
2023-05-17T09:10:35Z [ERROR] [/server/middleware/security/oidc_cli.go:62][requestID="8a4a009567f169bb3836ddfacc2bc1ac"]: failed to verify secret, username: admin, error: failed to get oidc user info, error: <QuerySeter> no row found
2023-05-17T09:10:35Z [DEBUG] [/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth

I asked a colleague to test this using docker CLI and he also has an error.

dioguerra commented 1 year ago

I have been trying to figure this out. This might help:

137.138.6.75 - - [23/May/2023:13:06:28 +0000] "GET /v2/ HTTP/1.1" 401 76 "-" "Go-http-client/1.1" 108 0.004 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 76 0.003 401 113419ecda19a3ed0751d061cb3594b8
137.138.6.75 - - [23/May/2023:13:06:28 +0000] "POST /service/token HTTP/1.1" 405 19 "-" "containerd/1.7.1+unknown" 361 0.004 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 19 0.003 405 4f2e550e6c8875aa9eee1226c8aebc30
137.138.6.75 - dtomasgu [23/May/2023:13:06:29 +0000] "GET /service/token?offline_token=true&service=harbor-registry HTTP/1.1" 200 635 "-" "containerd/1.7.1+unknown" 246 1.033 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 635 1.033 200 eb083294aca757ef9a88f14b48f61bc6
137.138.6.75 - - [23/May/2023:13:06:29 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "Go-http-client/1.1" 817 0.006 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 87 0.005 401 0c8bbad101876b337fd93e74123d2f57
137.138.6.75 - dtomasgu [23/May/2023:13:06:29 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "Go-http-client/1.1" 187 0.709 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 87 0.708 401 bd8cc6240778be5dc1c022629bcd0d1f
137.138.6.75 - dtomasgu [23/May/2023:13:06:30 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "Go-http-client/1.1" 187 0.735 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 87 0.735 401 6f3043764de1fab136d3c2fbcc5c2189
137.138.6.75 - dtomasgu [23/May/2023:13:06:31 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "Go-http-client/1.1" 187 0.695 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 87 0.695 401 9c49d806758a21e2e19fee984cf7f99b
137.138.6.75 - dtomasgu [23/May/2023:13:06:32 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "Go-http-client/1.1" 187 0.700 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 87 0.700 401 0c60a60bd59490943038e006d4cae3b7
137.138.6.75 - dtomasgu [23/May/2023:13:06:32 +0000] "GET /v2/ HTTP/1.1" 499 0 "-" "Go-http-client/1.1" 187 0.390 [registry-staging-harbor-core-80] [] 10.100.190.158:8080 0 0.390 - e278c7724c88e7b3959ca10b63a9d07b

dioguerra commented 1 year ago

The issue is mostly fixed. Just leave a note here for people with the same problem. This was a culmination of 3 issues affecting the same component:

Using argocd and not setting htpasswdString
Wrong htpasswdString name
Wrong generated htpasswd that was populated from the vault

dioguerra commented 1 year ago

This looks like an error where something is wrong. The Admin password should be the same as the portal one. Can you login into the portal with the Admin account?

Dmitrii Esin @.***> escreveu no dia segunda, 10/07/2023 à(s) 11:38:

@dioguerra https://github.com/dioguerra Hi! I have the same issue right now Deployed Harbor via ArgoCD, taking secrets from the Hashicorp Vault Tried to set htpasswdString (generated via htpasswd -b -c ./password username password) Not worked for me( Still getting

time="2023-07-10T09:36:24.573507058Z" level=error msg="error authenticating user "admin": authentication failure" go.version=go1.20.4 http.request.host=harbor.domain.name http.request.id=f20c16d9-e5da-4815-bfa1-8883bdf28d89 http.request.method=GET http.request.remoteaddr=10.5.30.91 http.request.uri="/v2/" http.request.useragent="docker/24.0.2 go/go1.20.4 git-commit/659604f kernel/5.15.49-linuxkit-pr os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.2 (darwin))" time="2023-07-10T09:36:24.574213755Z" level=warning msg="error authorizing context: basic authentication challenge for realm "harbor-registry-basic-realm": authentication failure" go.version=go1.20.4 http.request.host=harbor.domain.name http.request.id=f20c16d9-e5da-4815-bfa1-8883bdf28d89 http.request.method=GET http.request.remoteaddr=10.5.30.91 http.request.uri="/v2/" http.request.useragent="docker/24.0.2 go/go1.20.4 git-commit/659604f kernel/5.15.49-linuxkit-pr os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.2 (darwin))"

Can you share your thoughts? I appreciate you for your help!

— Reply to this email directly, view it on GitHub https://github.com/goharbor/harbor/issues/18480#issuecomment-1628595491, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZKHMKLIYXAEGT3DQQWBDXPPEQZANCNFSM6AAAAAAWTAJ6H4 . You are receiving this because you were mentioned.Message ID: @.***>

-- Diogo F. Tomás Guerra Master In Electronics Engineering Instituto Superior Técnico

(+41) 78 211 30 87 / (+351) 91 444 86 91 @.*** linkedin.com/in/diogoguerraist

EsDmitrii commented 1 year ago

yep, I solved this issue, thank you! I found that guys use this https://github.com/goharbor/harbor-helm/blob/master/templates/registry/registry-secret.yaml#L50 (b64enc inside the template) I'll prepare PR, we need to change secret from data: to stringData: and remove b64enc from all templates. My links to vault secrets templates before Argo and I get base64 encrypted link to Vault

goharbor / harbor

Error with authentication when using OIDC #18480

Debug

Jobservice logs show:

Core (not much use)