Failed to push artifacts during load

[kesslerdev]

Expected behavior and actual behavior:

When a load occurs (triggered by several push pipelines from GitLab) on Harbor, several pushes are successful in a decent amount of time (<10s), while others can hang for 10 to 30 minutes and fail. However, all pulls work.

We have scaled the core and registry from 3 to 6, with no significant improvements. Additionally, we don't have a very high load (maximum 40 operations per second for all API calls) with an average of 5 operations per second.

When the problem occurs, we can have 10 calls of PATCH /v2/project/image/blobs/uploads terminated with errors in over 600 seconds.

Versions: Please specify the versions of following systems.

• harbor version: [2.7.2]
• kubernetes version: [1.26.13]
• istio version: [1.18.3]

Additional context:

• When Harbor encounters issues with pushes:
  • The database is hardly processing any requests.
  • The S3 storage (backed by physical bay) used by Harbor and several applications (such as GitLab) does not show any problematic load, as "all indicators are green."
  • The built-in Redis is also idle.

Graphes from grafana: From Harbor: For v2_blob_upload PATCH & v2_blob_upload PUT

Also v2_blob_upload PATCH & v2_blob_upload PUT but for all core nodes

From istio:

Namespace pods:

NAME                                  READY   STATUS    RESTARTS       AGE
harbor-chartmuseum-7b479dd6d6-56m5g   1/1     Running   0              6d22h
harbor-core-6f7b48c949-btmw6          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-k8zh8          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-kx2gp          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-lhtqc          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-sww2m          1/1     Running   1 (5d6h ago)   5d6h
harbor-core-6f7b48c949-wzjc8          1/1     Running   1 (5d6h ago)   5d6h
harbor-exporter-5fc8975d8b-fhmjx      1/1     Running   0              21d
harbor-jobservice-544996775-cjnp9     1/1     Running   0              5d5h
harbor-nginx-df7bf48bc-xgw4n          1/1     Running   0              2d23h
harbor-nginx-df7bf48bc-zl9zs          1/1     Running   0              2d23h
harbor-portal-77786765fc-gvgt2        1/1     Running   0              6d22h
harbor-portal-77786765fc-j6kh2        1/1     Running   0              6d22h
harbor-portal-77786765fc-t7b97        1/1     Running   0              6d22h
harbor-redis-0                        1/1     Running   0              5d6h
harbor-registry-86579b6bcd-5jkbw      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-b25pz      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-f6wcg      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-gbrbj      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-kg2zt      2/2     Running   0              5d6h
harbor-registry-86579b6bcd-mh7wq      2/2     Running   0              5d6h
harbor-trivy-0                        1/1     Running   0              4d6h

• Harbor config files:
  • Installed by helm harbor chart harbor-1.11.2

    
    caBundleSecretName: harbor-crt
    caSecretName: harbor-crt
    cache:
    enabled: false
    chartmuseum:
    enabled: true
    core:
    configureUserSettings: '{"auth_mode":"oidc_auth","http_authproxy_admin_groups":"","http_authproxy_endpoint":"","http_authproxy_server_certificate":"","http_authproxy_skip_search":false,"http_authproxy_tokenreview_endpoint":"","http_authproxy_verify_cert":true,"notification_enable":true,"oidc_admin_group":"ADM_GRP","oidc_auto_onboard":true,"oidc_client_id":"CLIENTID","oidc_endpoint":"OIDC","oidc_extra_redirect_parms":{},"oidc_groups_claim":"GROUPS","oidc_name":"NAME","oidc_scope":"SCOPES","oidc_user_claim":"CLAIM","oidc_verify_cert":true,"project_creation_restriction":"adminonly","quota_per_project_enable":true,"read_only":false,"robot_name_prefix":"robot-","robot_token_duration":356,"self_registration":false,"storage_per_project":53687091200,"token_expiration":30,"oidc_client_secret":"SECRET"}'
    replicas: 3
    secretName: harbor-crt
    database:
    external:
    coreDatabase: registry
    host: PG_HOST
    notaryServerDatabase: notary_server
    notarySignerDatabase: notary_signer
    password: PG_PASSWORD
    port: "PG_PORT"
    sslmode: disable
    username: PG_USERNAME
    maxIdleConns: 100
    maxOpenConns: 900
    type: external
    enableMigrateHelmHook: false
    expose:
    ingress:
    hosts:
    core: hub.***.***.com
    tls:
    certSource: secret
    enabled: true
    secret:
    secretName: harbor-crt-tls
    type: ClusterIP
    externalURL: https://hub.***.***.com
    harborAdminPassword: ADMIN_PASSWORD
    internalTLS:
    enabled: false
    logLevel: info
    metrics:
    enabled: true
    serviceMonitor:
    additionalLabels:
    release: prom-app
    enabled: true
    notary:
    enabled: false
    persistence:
    enabled: true
    imageChartStorage:
    caBundleSecretName: harbor-crt
    disableredirect: true
    s3:
    accesskey: S3_ACCESSKEY
    bucket: S3_BUCKET
    encrypt: false
    region: us-east-1
    regionendpoint: S3_ENDPOINT
    rootdirectory: /
    secretkey: S3_SECRET
    secure: true
    v4auth: true
    type: s3
    persistentVolumeClaim:
    chartmuseum:
    accessMode: ReadWriteOnce
    size: 5Gi
    database:
    accessMode: ReadWriteOnce
    size: 1Gi
    jobservice:
    jobLog:
    accessMode: ReadWriteOnce
    size: 10Gi
    storageClass: rook
    redis:
    accessMode: ReadWriteOnce
    size: 5Gi
    storageClass: rook
    registry:
    accessMode: ReadWriteOnce
    size: 5Gi
    trivy:
    accessMode: ReadWriteOnce
    size: 5Gi
    storageClass: rook
    resourcePolicy: keep
    portal:
    replicas: 3
    proxy:
    components:

  • core
  • jobservice
  • trivy httpProxy: httpsProxy: noProxy: redis: type: internal registry: replicas: 3 secretKey: trace: enabled: false trivy: enabled: true

    
    [logs.zip](https://github.com/goharbor/harbor/files/14421349/logs.zip)

[chlins]

Could you please attach the harbor core and registry logs as well?

[kesslerdev]

Sure i've added the logs of core & registry pods in the issue

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.