search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
23.89k stars 4.74k forks source link

failed to verify certificate in one component of harbor with internal tls communication #18745

Closed bingo4933 closed 1 year ago

bingo4933 commented 1 year ago

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior: I'd like to deploy harbor with internal tls communication. I made theirs certificates and passed running ./prepare --with-notary --with-trivy and ./install.sh --with-notary --with-trivy

I could see every component goes well except harbor-jobservice endless restarting issue after running docker-compose ps The log message shown like

May 26 21:42:19 172.23.0.1 jobservice[14659]: 2023-05-26T13:42:19Z [ERROR] [/pkg/config/rest/rest.go:50]: Failed on load rest config err:Get "https://core:8443/api/v2.0/internalconfig": tls: failed to verify certificate: x509: certificate signed by unknown authority, url:https://core:8443/api/v2.0/internalconfig

could you please help to fix the issue or any misconfiguration? thanks a lot

Steps to reproduce the problem: Please provide the steps to reproduce this problem.

create certificate and provide to harbor and docker


$ openssl genrsa -out ca.key 4096
# Generate the CA certificate
$ openssl req -x509 -new -nodes -sha512 -days 3650 \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.local.com" \
-key ca.key \
-out ca.crt

Generate a Server Certificate

$ openssl genrsa -out registry.local.com.key 4096 $ openssl req -sha512 -new \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=registry.local.com" \ -key registry.local.com.key \ -out registry.local.com.csr $ vi v3.ext authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names

[alt_names] DNS.1=registry.local.com DNS.2=registry DNS.3=k8s-node3.local.com DNS.4=core DNS.5=10.138.181.6

$ openssl x509 -req -sha512 -days 3650 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in registry.local.com.csr \ -out registry.local.com.crt

copy certificate to harbor cert directory

$ cp registry.local.com.crt /harbor-data/cert/ $ cp registry.local.com.key /harbor-data/cert/

provide cert to docker

$ openssl x509 -inform PEM -in registry.local.com.crt -out registry.local.com.cert $ cp registry.local.com.cert /etc/docker/certs.d/registry.local.com/ $ cp registry.local.com.key /etc/docker/certs.d/registry.local.com/ $ cp ca.crt /etc/docker/certs.d/registry.local.com/

> create tls cert
```bash
# copy ca cert to tls cert directory
$ cp ca.crt ca.key v3.ext /harbor-data/tls/cert/
$ cd /harbor-data/tls/cert/

# create tls cert for each component, here take jobservice as example
$ openssl genrsa -out job_service.key 4096

$ openssl req -sha512 -new \
-subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=jobservice" \
-key job_service.key \
-out job_service.csr

$ openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in job_service.csr \
-out job_service.crt

here are all cert in /harbor-data/tls/cert/

$ ls /harbor-data/tls/cert/
ca.crt           clair.crt  harbor_db.crt           job_service.crt    notary_signer.crt  proxy.crt        registryctl.csr    v3.ext
ca.key           clair.csr  harbor_db.csr           job_service.csr    notary_signer.csr  proxy.csr        registryctl.key
ca.srl           clair.key  harbor_db.key           job_service.key    notary_signer.key  proxy.key        registry.key
chartmuseum.crt  core.crt   harbor_internal_ca.crt  notary_server.crt  portal.crt         registry.crt     trivy_adapter.crt
chartmuseum.csr  core.csr   harbor_internal_ca.csr  notary_server.csr  portal.csr         registry.csr     trivy_adapter.csr
chartmuseum.key  core.key   harbor_internal_ca.key  notary_server.key  portal.key         registryctl.crt  trivy_adapter.key

launch harbor with internal tls communication


$ ./prepare --with-notary --with-trivy
prepare base dir is set to /root/harbor
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/conf.d/notary.upstream.conf
Clearing the configuration file: /config/nginx/conf.d/notary.server.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Clearing the configuration file: /config/shared/trust-certificates/harbor_internal_ca.crt
Clearing the configuration file: /config/shared/trust-certificates/core.crt
Clearing the configuration file: /config/notary/server-config.postgres.json
Clearing the configuration file: /config/notary/server_env
Clearing the configuration file: /config/notary/signer_env
Clearing the configuration file: /config/notary/signer-config.postgres.json
Clearing the configuration file: /config/notary/notary-signer.key
Clearing the configuration file: /config/notary/notary-signer.crt
Clearing the configuration file: /config/notary/root.crt
Clearing the configuration file: /config/notary/notary-signer-ca.crt
Clearing the configuration file: /config/trivy-adapter/env
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Successfully called func: create_root_cert
Successfully called func: create_cert
Copying certs for notary signer
Copying nginx configuration file for notary
Generated configuration file: /config/nginx/conf.d/notary.upstream.conf
Generated configuration file: /config/nginx/conf.d/notary.server.conf
Generated configuration file: /config/notary/server-config.postgres.json
Generated configuration file: /config/notary/server_env
Generated and saved secret to file: /data/secret/keys/defaultalias
Generated configuration file: /config/notary/signer_env
Generated configuration file: /config/notary/signer-config.postgres.json
Generated configuration file: /config/trivy-adapter/env
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

$ ./install.sh --with-notary --with-trivy

[Step 0]: checking if docker is installed ...

Note: docker version: 19.03.14

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 2.18.1

[Step 2]: loading Harbor images ... Loaded image: goharbor/registry-photon:v2.8.1 Loaded image: goharbor/notary-server-photon:v2.8.1 Loaded image: goharbor/harbor-core:v2.8.1 Loaded image: goharbor/harbor-jobservice:v2.8.1 Loaded image: goharbor/harbor-exporter:v2.8.1 Loaded image: goharbor/trivy-adapter-photon:v2.8.1 Loaded image: goharbor/prepare:v2.8.1 Loaded image: goharbor/redis-photon:v2.8.1 Loaded image: goharbor/nginx-photon:v2.8.1 Loaded image: goharbor/notary-signer-photon:v2.8.1 Loaded image: goharbor/harbor-portal:v2.8.1 Loaded image: goharbor/harbor-log:v2.8.1 Loaded image: goharbor/harbor-db:v2.8.1 Loaded image: goharbor/harbor-registryctl:v2.8.1

[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ... prepare base dir is set to /root/harbor Clearing the configuration file: /config/portal/nginx.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/log/rsyslog_docker.conf Clearing the configuration file: /config/nginx/conf.d/notary.upstream.conf Clearing the configuration file: /config/nginx/conf.d/notary.server.conf Clearing the configuration file: /config/nginx/nginx.conf Clearing the configuration file: /config/core/env Clearing the configuration file: /config/core/app.conf Clearing the configuration file: /config/registry/passwd Clearing the configuration file: /config/registry/config.yml Clearing the configuration file: /config/registryctl/env Clearing the configuration file: /config/registryctl/config.yml Clearing the configuration file: /config/db/env Clearing the configuration file: /config/jobservice/env Clearing the configuration file: /config/jobservice/config.yml Clearing the configuration file: /config/shared/trust-certificates/harbor_internal_ca.crt Clearing the configuration file: /config/notary/server-config.postgres.json Clearing the configuration file: /config/notary/server_env Clearing the configuration file: /config/notary/signer_env Clearing the configuration file: /config/notary/signer-config.postgres.json Clearing the configuration file: /config/trivy-adapter/env Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml loaded secret from file: /data/secret/keys/secretkey Copying nginx configuration file for notary Generated configuration file: /config/nginx/conf.d/notary.upstream.conf Generated configuration file: /config/nginx/conf.d/notary.server.conf Generated configuration file: /config/notary/server-config.postgres.json Generated configuration file: /config/notary/server_env loaded secret from file: /data/secret/keys/defaultalias Generated configuration file: /config/notary/signer_env Generated configuration file: /config/notary/signer-config.postgres.json Generated configuration file: /config/trivy-adapter/env Generated configuration file: /compose_location/docker-compose.yml Clean up the input dir

Note: stopping existing Harbor instance ...

[Step 5]: starting Harbor ... ➜ Notary will be deprecated as of Harbor v2.6.0 and start to be removed in v2.8.0 or later. You can use cosign for signature instead since Harbor v2.5.0. Please see discussion here for more details. https://github.com/goharbor/harbor/discussions/16612 [+] Building 0.0s (0/0)
[+] Running 15/15 ✔ Network harbor_harbor-notary Created
✔ Network harbor_notary-sig Created
✔ Network harbor_harbor Created
✔ Container harbor-log Started
✔ Container redis Started
✔ Container registryctl Started
✔ Container harbor-db Started
✔ Container registry Started
✔ Container harbor-portal Started
✔ Container notary-signer Started
✔ Container trivy-adapter Started
✔ Container harbor-core Started
✔ Container notary-server Started
✔ Container harbor-jobservice Started
✔ Container nginx Started
✔ ----Harbor has been installed and started successfully.----

$ docker-compose ps | grep Restarting harbor-jobservice goharbor/harbor-jobservice:v2.8.1 "/harbor/entrypoint.…" jobservice 3 hours ago Restarting (2) 53 seconds ago


**Versions:**
Please specify the versions of following systems.
- harbor version: [v2.8.1]
- docker engine version: [Docker version 19.03.14, build 5eb3275d40]
- docker-compose version: [v2.18.1]

**Additional context:**

- **Harbor config files:** 
```yaml
$ egrep -v '^#|^$|  #' harbor.yml
hostname: registry.local.com
http:
  port: 80
https:
  port: 443
  certificate: /harbor-data/cert/registry.local.com.crt
  private_key: /harbor-data/cert/registry.local.com.key
internal_tls:
  enabled: true
  dir: /harbor-data/tls/cert
harbor_admin_password: abcd
database:
  password: root123
  max_idle_conns: 100
  max_open_conns: 900
  conn_max_lifetime: 5m
  conn_max_idle_time: 0
data_volume: /harbor-data/db-data
trivy:
  ignore_unfixed: false
  skip_update: false
  offline_scan: false
  security_check: vuln
  insecure: false
jobservice:
  max_job_workers: 10
  logger_sweeper_duration: 1 #days
notification:
  webhook_job_max_retry: 3
  webhook_job_http_client_timeout: 3 #seconds
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
_version: 2.8.0
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy
upload_purging:
  enabled: true
  age: 168h
  interval: 24h
  dryrun: false
cache:
  enabled: false
  expire_hours: 24
github-actions[bot] commented 1 year ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] commented 1 year ago

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.