Harbor OIDC client secret missing after restart/re-deployment

[chhabriv]

I am using harbor helm to manage the K8s deployment of harbor v2.8.0 Post deployment, the goharbor/harbor terraform provider v3.10.2 is used to setup the OIDC auth mode for the harbor deployment.

This works correctly, until the harbor app is redeployed or pods are replaced. Once the harbor pods are re-created, it appears harbor no longer send the client secret in the OIDC request and as a result we get 401 from our OIDC provider.

Generating a new secret and updating it fixes the issue, but requires frequent manual intervention

[zyyw]

could you please share with us the core log

[chhabriv]

I think I've been able to narrow down the issue a bit. This seems to happen when harbor is redeployed (helm upgrade ...). Even though there are no changes to the app/chart, some of the credentials (like core.secret) get regenerated on each run causing an upgrade. After this, harbor seems to lose the previously set oidc client secret. PFA below logs from core after an upgrade, and the failure of the first oidc login request. The oidc provider returns a 401 because of missing auth (client secret).

Logs:

Appending internal tls trust CA to ca-bundle ... find: '/etc/harbor/ssl': No such file or directory Internal tls trust CA appending is Done. 2023/11/08 11:42:54.027 [D] init global config instance failed. If you do not use this, just ignore it. open conf/app.conf: no such file or directory 2023-11-08T11:42:54Z [INFO] [/controller/artifact/annotation/parser.go:71]: the annotation parser to parser artifact annotation version v1alpha1 registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.wasm.config.v1+json registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cncf.helm.config.v1+json registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.cnab.manifest.v1 registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.oci.image.index.v1+json registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.distribution.manifest.list.v2+json registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.distribution.manifest.v1+prettyjws registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.oci.image.config.v1+json registered 2023-11-08T11:42:54Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.docker.container.image.v1+json registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/native/adapter.go:36]: the factory for adapter docker-registry registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/aliacr/adapter.go:30]: the factory for adapter ali-acr registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/awsecr/adapter.go:44]: the factory for adapter aws-ecr registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/azurecr/adapter.go:29]: Factory for adapter azure-acr registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/dockerhub/adapter.go:26]: Factory for adapter docker-hub registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/dtr/adapter.go:22]: the factory of dtr adapter was registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/githubcr/adapter.go:29]: the factory for adapter github-ghcr registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/gitlab/adapter.go:18]: the factory for adapter gitlab registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/googlegcr/adapter.go:37]: the factory for adapter google-gcr registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/harbor/adaper.go:31]: the factory for adapter harbor registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/huawei/huawei_adapter.go:40]: the factory of Huawei adapter was registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/jfrog/adapter.go:42]: the factory of jfrog artifactory adapter was registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/quay/adapter.go:53]: the factory of Quay adapter was registered 2023-11-08T11:42:54Z [INFO] [/pkg/reg/adapter/tencentcr/adapter.go:41]: the factory for adapter tencent-tcr registered 2023-11-08T11:42:54Z [INFO] [/core/controllers/base.go:159]: Config path: /etc/core/app.conf 2023-11-08T11:42:54Z [INFO] [/core/main.go:141]: initializing cache ... 2023-11-08T11:42:54Z [INFO] [/core/main.go:151]: initializing configurations... 2023-11-08T11:42:54Z [INFO] [/lib/config/systemconfig.go:187]: key path: /etc/core/key 2023-11-08T11:42:54Z [INFO] [/lib/config/config.go:92]: init secret store 2023-11-08T11:42:54Z [INFO] [/core/main.go:153]: configurations initialization completed 2023-11-08T11:42:54Z [INFO] [/common/dao/base.go:67]: Registering database: type-PostgreSQL XXXXXXXXXXXXX database-registry sslmode-"require" 2023-11-08T11:42:54Z [INFO] [/lib/metric/server.go:23]: Prometheus metric server running on port 9090 2023-11-08T11:42:54Z [INFO] [/common/dao/base.go:72]: Register database completed 2023-11-08T11:42:54Z [INFO] [/common/dao/pgsql.go:135]: Upgrading schema for pgsql ... 2023-11-08T11:42:54Z [INFO] [/common/dao/pgsql.go:138]: No change in schema, skip. 2023-11-08T11:42:54Z [INFO] [/migration/migration.go:60]: Abstracting artifact data to DB... 2023-11-08T11:42:54Z [INFO] [/migration/migration.go:69]: No need to abstract artifact data. Skip 2023-11-08T11:42:54Z [INFO] [/core/main.go:185]: The database has been migrated successfully 2023-11-08T11:42:54Z [INFO] [/core/main.go:93]: User id: 1 already has its encrypted password. 2023-11-08T11:42:54Z [INFO] [/core/main.go:311]: Removing Trivy scanner 2023-11-08T11:42:54Z [ERROR] [/pkg/audit/forward.go:44]: failed to create audit log, error dial tcp: missing address 2023-11-08T11:42:54Z [INFO] [/core/main.go:223]: initializing notification... 2023-11-08T11:42:54Z [INFO] [/pkg/notification/notification.go:63]: notification initialization completed 2023-11-08T11:42:54Z [INFO] [/core/main.go:242]: Version: v2.8.0, Git commit: 89ef156d 2023-11-08T11:42:54Z [INFO] [/core/main.go:244]: Fix empty subiss for meta info data. 2023-11-08T11:42:54Z [INFO] [/pkg/oidc/fix.go:23]: Not found any records with empty subiss, good to go. 2023/11/08 11:42:54.528 [I] [server.go:281] http server Running on http://:8080 2023-11-08T11:44:06Z [INFO] [/lib/encrypt/encrypt.go:60]: the path of key used by key provider: /etc/core/key 2023-11-08T11:44:41Z [INFO] [/controller/registry/controller.go:222]: Start regular health check for registries with interval 5m0s 2023-11-08T11:47:31Z [ERROR] [/core/controllers/oidc.go:112]: Failed to exchange token, error: oauth2: cannot fetch token: 401 Unauthorized

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.