search

goharbor / harbor-helm

The helm chart to deploy Harbor
Apache License 2.0
1.19k stars 760 forks source link

Incorrect generation of harbor-core URL in part of jobservice #1790

Closed Tonkari closed 3 weeks ago

Tonkari commented 4 months ago

Summary

Jobservice logs show that some calls from hook_agent go to an incorrect harbor-core URL

Environment

We are using this chart to deploy harbor 2.10.2 (chart version 1.14.2) with internal TLS enabled, certSource is secret and the secrets are managed by cert-manager. This means our services all listen exclusively on port 443

Problem

When troubleshooting an unrelated issue with the jobservice we noticed some errors in the pod logs that contain failed requests to "http://harbor-core:80", when all other requests from the pod go to the correct URL of "https://harbor-core:443"

Log messages

# failed call
2024-07-11T07:57:13Z [ERROR] [/jobservice/hook/hook_agent.go:159]: Retry: still failed after all retries: Post "http://harbor-core:80/service/notifications/tasks/xxx": context deadline exceeded (Client.Timeout exceeded while awaiting headers), evt=status change: job=xxxxxxxxxxxxxxxxxxxxxxxx@1720656000, status=Success, revision=1720655940->http://harbor-core:80/service/notifications/tasks/xxx

# working call from same pod
2024-07-11T08:00:00Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://harbor-core:443/api/v2.0/internalconfig

I don't know if it's a configuration error in this chart of if it's hardcoded in the docker image itself - I would appreciate any help or direction. I also don't know what functions of harbor are impaired here or what earlier versions are affected.

zyyw commented 3 months ago

@Tonkari thanks for reporting this issue. Could you please share with us what triggers the # failed call.

Tried configuring a webhook for image scanning on an internalTLS enabled Harbor (harbor-helm main branch / harbor oss main branch), do not see the http://harbor-core:80 error log. 

2024-07-25T08:28:11Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"IMAGE_SCAN","id":"fe263f2eddb3492c72786b57","t":1721896089,"args":null}
2024-07-25T08:28:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:16Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f57d80b348d3711a7be26fc8","t":1721896091,"args":null}
2024-07-25T08:28:16Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:16Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:17Z [INFO] [/jobservice/runner/redis.go:152]: Job 'WEBHOOK:f57d80b348d3711a7be26fc8' exit with success
2024-07-25T08:28:31Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"8792a36d444c0a213f42259f","t":1721896110,"args":null}
2024-07-25T08:28:31Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:31Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:32Z [INFO] [/jobservice/runner/redis.go:152]: Job 'WEBHOOK:8792a36d444c0a213f42259f' exit with success
2024-07-25T08:28:33Z [INFO] [/pkg/scan/postprocessors/report_converters.go:196][report="bdf79a35-1494-42ed-9e18-34b394573625" scanner="e18e636d-4a5e-11ef-b796-7a71a6a03a13" vulnerabilityRecords="20"]: Converted vulnerability records to the new schema
2024-07-25T08:28:33Z [INFO] [/jobservice/runner/redis.go:152]: Job 'IMAGE_SCAN:fe263f2eddb3492c72786b57' exit with success
2024-07-25T08:28:33Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f7c4c07113cd5bd13ac3e05a","t":1721896113,"args":null}
2024-07-25T08:28:33Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:33Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:34Z [INFO] [/jobservice/runner/redis.go:152]: Job 'WEBHOOK:f7c4c07113cd5bd13ac3e05a' exit with success
zyyw commented 3 months ago

Also tried replication with webhook on an internalTLS enabled Harbor (v2.10.2, harbor-helm 1.14.2). Not seeing http://harbor-core:80 error log either. 

2024-07-25T09:23:58Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"REPLICATION","id":"6e366f8623a6ca639f078b45","t":1721899436,"args":null}
2024-07-25T09:23:58Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:23:58Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"d45b9170f1c79f961efad041","t":1721899438,"args":null}
2024-07-25T09:24:00Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:00Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:00Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:d45b9170f1c79f961efad041' exit with success
2024-07-25T09:24:02Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"e09e1552894eb181a6a3a408","t":1721899442,"args":null}
2024-07-25T09:24:02Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:02Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:02Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:e09e1552894eb181a6a3a408' exit with success
2024-07-25T09:24:03Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"473e584319cd5a939f05eba5","t":1721899443,"args":null}
2024-07-25T09:24:03Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:03Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:04Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:473e584319cd5a939f05eba5' exit with success
2024-07-25T09:24:04Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f870f187b6595097bc5851ed","t":1721899444,"args":null}
2024-07-25T09:24:04Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:04Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:05Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:f870f187b6595097bc5851ed' exit with success
2024-07-25T09:24:06Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"1252722613e2105462c6389d","t":1721899446,"args":null}
2024-07-25T09:24:06Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:06Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:07Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:1252722613e2105462c6389d' exit with success
2024-07-25T09:24:07Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"204a55fdd1104289c763a615","t":1721899447,"args":null}
2024-07-25T09:24:07Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:07Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:09Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:204a55fdd1104289c763a615' exit with success
2024-07-25T09:24:09Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"a0b7acddbf04580fcd61abec","t":1721899448,"args":null}
2024-07-25T09:24:09Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:09Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:10Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"5b2286920779f2e22770f43a","t":1721899450,"args":null}
2024-07-25T09:24:10Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:10Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:10Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:a0b7acddbf04580fcd61abec' exit with success
2024-07-25T09:24:11Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"58c58549b55cd43089d1a0c9","t":1721899451,"args":null}
2024-07-25T09:24:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:12Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:5b2286920779f2e22770f43a' exit with success
2024-07-25T09:24:13Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:58c58549b55cd43089d1a0c9' exit with success
2024-07-25T09:24:13Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"7d7f06d63a55d227d81c9a1c","t":1721899452,"args":null}
2024-07-25T09:24:13Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:13Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:7d7f06d63a55d227d81c9a1c' exit with success
2024-07-25T09:24:14Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f711fcdc772a57aa69406af5","t":1721899453,"args":null}
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/jobservice/runner/redis.go:151]: Job 'REPLICATION:6e366f8623a6ca639f078b45' exit with success
2024-07-25T09:24:14Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"145b9765575f365b7bcc789d","t":1721899454,"args":null}
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:f711fcdc772a57aa69406af5' exit with success
2024-07-25T09:24:14Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"7b9c0397e795f064f85b2718","t":1721899454,"args":null}
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:15Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:7b9c0397e795f064f85b2718' exit with success
2024-07-25T09:24:15Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:145b9765575f365b7bcc789d' exit with success
chlins commented 3 months ago

@Tonkari If you initially used port 80 and later switched to 443, there might be some old tasks that send requests to the old address. Please try restarting the harbor core and jobservice containers, and check their environment variables for CORE_URL related configurations.

Tonkari commented 3 months ago

Hi, thanks for the investigation,

We were indeed using port 80 initially, but that was many years and versions ago. In the meantime all pods have been updated and restarted multiple times (we are running everything with at least 3 replicas), and a manual restart of jobservice did not get rid of the error. How can I identify the job that triggers the notification? We are not running any replication between other registries but everything else should be configured (GC, scans with built-in trivy, retention mostly). It always calls the same URL as well http://harbor-core:80/service/notifications/tasks/186, not multiple different ones.

github-actions[bot] commented 1 month ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] commented 3 weeks ago

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.