Harbor core crashes on editing nginx.conf

harshguptaserver commented 2 weeks ago

ask: edit nginx.conf file to add oscp support

ssl_stapling on;
  ssl_stapling_verify on;
  resolver 1.1.1.1 1.0.0.1 [2606:4700:4700::1111] [2606:4700:4700::1001] valid=300s;
  resolver_timeout 10s;
  ssl_trusted_certificate  /etc/cert/server.crt;

I am using letsencrypt signed certificate in harbor.yml , the certificate works fine in google chrome browser but for firefox and others i get TLS warning ( it seems something to do with ocsp stapling )

in order to get rid of the error i need to edit the nginx.conf file in common folder and to do that i do the following:
```
docker-compose down
sudo vi common/config/nginx/nginx.conf 
docker compose up
```

if i do docker compose down and docker compose up without editing nginx file , it all works fine.

issue

as soon as i restart docker compose after making 1 line of changes in nginx , harbour-core starts crashing

CONTAINER ID   IMAGE                                  COMMAND                  CREATED         STATUS                            PORTS                                                                                NAMES
92fb4341fd8e   goharbor/nginx-photon:v1.10.19         "nginx -g 'daemon of…"   8 seconds ago   Up 6 seconds (health: starting)   0.0.0.0:80->8080/tcp, [::]:80->8080/tcp, 0.0.0.0:443->8443/tcp, [::]:443->8443/tcp   nginx
7058192a20fe   goharbor/harbor-jobservice:v1.10.19    "/harbor/harbor_jobs…"   8 seconds ago   Up 6 seconds (health: starting)                                                                                        harbor-jobservice
9208c44361cc   goharbor/harbor-core:v1.10.19          "/harbor/harbor_core"    8 seconds ago   Restarting (1) 1 second ago

On checking the harbor core container logs it seems to have password issues, which makes no sense as i only added 1 line in nginx and restarted ( docker compose down ; edit nginx ; docker compose up)

2024-11-05T13:28:14Z [INFO] [/core/config/config.go:100]: key path: /etc/core/key
2024-11-05T13:28:14Z [ERROR] [/common/config/manager.go:118]: loadSystemConfigFromEnv failed, config item, key: clair_db_port,  err: strconv.Atoi: parsing "": invalid syntax
2024-11-05T13:28:14Z [INFO] [/core/config/config.go:73]: init secret store
2024-11-05T13:28:14Z [INFO] [/core/config/config.go:76]: init project manager based on deploy mode
2024-11-05T13:28:14Z [INFO] [/core/config/config.go:145]: initializing the project manager based on local database...
2024-11-05T13:28:14Z [INFO] [/core/main.go:181]: configurations initialization completed
2024-11-05T13:28:14Z [INFO] [/common/dao/base.go:84]: Registering database: type-PostgreSQL host-postgresql port-5432 databse-registry sslmode-"disable"
[ORM]2024/11/05 13:28:14 register db Ping `default`, pq: password authentication failed for user "postgres"
2024-11-05T13:28:14Z [FATAL] [/core/main.go:188]: failed to initialize database: register db Ping `default`, pq: password authentication failed for user "postgres"

Please help clarify how to edit nginx.conf safely and avoid harbor-core crashing

harshguptaserver commented 2 weeks ago

Add the issue is strange because even after editing the nginx.conf, nginx container and ui works, since core is crashing login fails and everything is down:

root@server:/home/pi/harbour/harbor# docker ps
CONTAINER ID   IMAGE                                  COMMAND                  CREATED         STATUS                                 PORTS                                                                                NAMES
82459a7b7189   goharbor/nginx-photon:v1.10.19         "nginx -g 'daemon of…"   7 minutes ago   Up About a minute (healthy)            0.0.0.0:80->8080/tcp, [::]:80->8080/tcp, 0.0.0.0:443->8443/tcp, [::]:443->8443/tcp   nginx
39d82f220b2e   goharbor/harbor-jobservice:v1.10.19    "/harbor/harbor_jobs…"   7 minutes ago   Up 11 seconds (health: starting)                                                                                            harbor-jobservice
b57cffb54008   goharbor/harbor-core:v1.10.19          "/harbor/harbor_core"    7 minutes ago   Restarting (1) 25 seconds ago                                                                                               harbor-core
246fc8051956   goharbor/harbor-portal:v1.10.19        "nginx -g 'daemon of…"   7 minutes ago   Up About a minute (healthy)            8080/tcp                                                                             harbor-portal
dc117ad9e316   goharbor/harbor-registryctl:v1.10.19   "/home/harbor/start.…"   7 minutes ago   Up About a minute (healthy)                                                                                                 registryctl
8184e5413183   goharbor/registry-photon:v1.10.19      "/home/harbor/entryp…"   7 minutes ago   Up About a minute (healthy)            5000/tcp                                                                             registry
8e48cc538cc8   goharbor/harbor-db:v1.10.19            "/docker-entrypoint.…"   7 minutes ago   Up About a minute (health: starting)   5432/tcp                                                                             harbor-db
4337c0912136   goharbor/redis-photon:v1.10.19         "redis-server /etc/r…"   7 minutes ago   Up About a minute (healthy)            6379/tcp                                                                             redis
de1c34bd741c   goharbor/harbor-log:v1.10.19           "/bin/sh -c /usr/loc…"   7 minutes ago   Up About a minute (healthy)            127.0.0.1:1514->10514/tcp                                                            harbor-log

ui is working i.e nginx container is up

root@server:/home/pi/harbour/harbor# curl https://container-registry.example.com -k
<!doctype html>
<html>

<head>
    <meta charset="utf-8">
    <title>Harbor</title>
    <base href="/">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
<link rel="stylesheet" href="styles.701dc5ee3007bd83bfa4.css"></head>

<body>
    <harbor-app>
        <div class="spinner spinner-lg app-loading">
            Loading...
        </div>
    </harbor-app>
<script src="runtime.9ad22a88fcc70a015907.js" defer></script><script src="polyfills-es5.d01e8ad6bc0c07b49ab6.js" nomodule defer></script><script src="scripts.7fa3fa51e1a86dfba2c8.js" defer></script><script src="main.7d1007223a0d5be240a8.js" defer></script></body>

wy65701436 commented 2 weeks ago

hi, please try your scenario with latest harbor release v2.11.1.

MinerYang commented 2 weeks ago

Hi @harshguptaserver ,

If you are using self signed certificate, it may having warning on several browsers
Please make sure the ssl certificate and key were placed in the proper place like
```
# SSL
ssl_certificate /etc/cert/server.crt;
ssl_certificate_key /etc/cert/server.key;
```
- The db error may related the data volume directory been corrupted, please clean it up(it will lost all the harbor data if your don't mind) and then re-install as a total fresh one

harshguptaserver commented 2 weeks ago

Solved,

Amazing @wy65701436 thank you :-) i mistakenly was trying the older version.

Used 2.11.1
docker compose down

then editing the nginx.conf for ocsp as follows (to fix MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING ):

ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 1.0.0.1 [2606:4700:4700::1111] [2606:4700:4700::1001] valid=300s;
resolver_timeout 10s;
ssl_trusted_certificate  /etc/cert/server.crt;

docker compose up

and everything is working.

Thank you once again @MinerYang / @wy65701436 for the quick responses and helping figure this out.

goharbor / harbor