search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.14k stars 4.76k forks source link

[ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error #20819

Open rajatrj16 opened 3 months ago

rajatrj16 commented 3 months ago

I am occasionally getting the below error and due to this not able to push the image unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

Harbor version: Version v2.11.0-70255684 harbor-helm: v1.15.0

Harbor core is constantly giving below errors. There is no error in the istio-proxy 

Appending internal tls trust CA to ca-bundle ...
find: '/etc/harbor/ssl': No such file or directory
Internal tls trust CA appending is Done.
2024/08/05 06:24:43.594 [D]  init global config instance failed. If you do not use this, just ignore it.  open conf/app.conf: no such file or directory
2024-08-05T06:24:43Z [ERROR] [/lib/cache/cache.go:124]: failed to ping redis://harbor-redis.management.eks.region.aws.client.cloud:6379/0?idle_timeout_seconds=30, retry after 500ms : dial tcp 10.39.xx.xx:6379: connect: connection refused
2024-08-05T06:24:44Z [ERROR] [/pkg/audit/forward.go:44]: failed to create audit log, error dial tcp: missing address
2024/08/05 06:24:44.494 [I] [server.go:281]  http server Running on http://:8080
Appending internal tls trust CA to ca-bundle ...
find: '/etc/harbor/ssl': No such file or directory
Internal tls trust CA appending is Done.
2024/08/05 06:25:01.515 [D]  init global config instance failed. If you do not use this, just ignore it.  open conf/app.conf: no such file or directory
2024-08-05T06:25:01Z [ERROR] [/lib/cache/cache.go:124]: failed to ping  #redis://harbor-redis.management.eks.region.aws.client.cloud:6379/0?idle_timeout_seconds=30, retry after 500ms : dial tcp 10.39.xx.xx:6379: connect: connection refused
2024-08-05T06:25:02Z [ERROR] [/pkg/audit/forward.go:44]: failed to create audit log, error dial tcp: missing address
2024/08/05 06:25:02.341 [I] [server.go:281]  http server Running on http://:8080
2024-08-05T06:25:21Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:23Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:23Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:23Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T06:25:30Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:08:42Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:08:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:43Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:44Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:44Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:09:54Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:41Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:41Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024/08/05 07:14:41 http: proxy error: context canceled
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:47Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:58Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:14:58Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024/08/05 07:14:58 http: proxy error: context canceled
2024/08/05 07:14:58 http: proxy error: context canceled
2024/08/05 07:14:58 http: proxy error: context canceled
2024-08-05T07:15:08Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:15:08Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:23:25Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error
2024-08-05T07:23:25Z [ERROR] [/pkg/token/token.go:81]: parse token error, token signature is invalid: crypto/rsa: verification error

Jobservice logs: 

2024-08-05T06:25:43Z [ERROR] [/pkg/config/rest/rest.go:50]: Failed on load rest config err:http error: code 401, message {"errors":[{"code":"UNAUTHORIZED","message":"only internal service is allowed to call this API"}]}, url:http://harbor-registry-core:80/api/v2.0/internalconfig
2024-08-05T08:00:00Z [ERROR] [/pkg/config/manager.go:81]: AllConfigs failed, error failed to load rest config
2024-08-05T08:00:10Z [ERROR] [/jobservice/runtime/bootstrap.go:168]: trigger hook event error: Post "https://harbor-registry-core:443/service/notifications/tasks/2": read tcp 100.xx.xx.37:48090->172.20.xx.xx:443: read: connection reset by peer
2024-08-05T06:25:43Z [ERROR] [/jobservice/job/impl/context.go:76]: Job context initialization error: failed to load rest config
2024-08-05T08:02:26Z [ERROR] [/jobservice/hook/hook_agent.go:155]: Retry: sending hook event error: Post "https://harbor-registry-core:443/service/notifications/tasks/2": read tcp 100.64.xx.xx:39476->172.xx.xx.255:443: read: connection reset by peer, evt=status change: job=a7f29fa10abe9e99b13bd6f3@1722844800, status=Running, revision=1722844704, check_in=checkin->https://harbor-registry-core:443/service/notifications/tasks/2, duration=1m18.436330113s
zyyw commented 3 months ago

Is this a fresh install or upgrade from an older harbor? Also what is your authentication method, basic auth / oidc / ldap?

Found this issue might be related. Maybe you want to take a look at it:

rajatrj16 commented 3 months ago

Is this a fresh install or upgrade from an older harbor? Also what is your authentication method, basic auth / oidc / ldap?

Found this issue might be related. Maybe you want to take a look at it:

This was an upgrade from 2.10 to 2.11.0. The auth method is OIDC.

I have not configured any credentials except existingSecretAdminPassword and existingSecretAdminPasswordKey in values.yaml but I get unauthorized errors occasionally for robot accounts.

NikolaiBessonov commented 2 months ago

We have the same issue. It's floating problem. The problem reproduced in May of 2024, 09 of September and today again. Harbor version - 2.11.0 with s3 storage.

UPD: I have analyzed logs and metrics and got new information. When, core component runs at few replicas(e.g. 3), and controller deletes one of that replicas, new pod starts faster, then old goes delete. After new replica says in logs "Server running at 0.0.0.0:8080" errors appear.

github-actions[bot] commented 1 day ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.