goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.05k stars 4.75k forks source link

Updating database. Failed to connect DB after 30 seconds, time out on Notary server and Signer. #15898

Closed netanbu closed 2 years ago

netanbu commented 3 years ago

Hi,

I did setup harbor HA with external posgres, redis. All pods are sucessfully running. But notary pods were stuck CrashLoopBackOff state. Could you please advise what could be the error ? The logs of the both notary pods shows.

Updating database. Failed to connect DB after 30 seconds, time out

PFB pod status. kubectl get pods -n harbor

NAME READY STATUS RESTARTS AGE harbor-release1-chartmuseum-7bbbf46d44-trlvb 1/1 Running 0 6h15m harbor-release1-core-6577b7fdbd-xrfvz 1/1 Running 0 104m harbor-release1-jobservice-54d66dd554-zkr5p 1/1 Running 0 101m harbor-release1-notary-server-7df94cdd4b-5mlgb 0/1 CrashLoopBackOff 47 4h3m harbor-release1-notary-signer-ff6dcfd68-mg6n9 0/1 CrashLoopBackOff 47 4h3m harbor-release1-portal-b9d7947c4-s29gz 1/1 Running 0 110m harbor-release1-registry-584fcdcc94-g4nhr 2/2 Running 0 109m harbor-release1-trivy-0 1/1 Running 0 6h14m

kubectl get pods -n pgo ==> posgres NAME READY STATUS RESTARTS AGE backrest-backup-pgo4harbor-lw9vg 0/1 Completed 0 6h13m pgo-deploy-rdq5r 0/1 Completed 0 5d18h pgo4harbor-575c8fdc79-9mk8d 1/1 Running 0 6h18m pgo4harbor-backrest-shared-repo-7d6ddb4446-fzszn 1/1 Running 0 6h19m pgo4harbor-pgadmin-6b85cf6b4c-5g24j 1/1 Running 0 6h17m postgres-operator-6b48fbc879-2k6m4 4/4 Running 144 5d18h

kubectl get pods -n redis

NAME READY STATUS RESTARTS AGE haproxy-deployment-8bf7665cc-6kz94 1/1 Running 2 27h haproxy-deployment-8bf7665cc-9wvcc 1/1 Running 2 27h redis-0 1/1 Running 0 6h19m redis-1 1/1 Running 0 6h17m redis-2 1/1 Running 0 6h16m sentinel-0 1/1 Running 0 6h19m sentinel-1 1/1 Running 0 6h16m sentinel-2 1/1 Running 0 6h16m

wy65701436 commented 3 years ago

what's the Harbor & helm version? Please provide the log as well. BTW, you can file helm issue on the helm repo.

tribock commented 2 years ago

same issue here: harbor version: v2.4.1 postgres version: 13 operator: crunchy patroni: dynamicConfiguration: postgresql: pg_hba:

LOG:

2022/01/26 09:44:26 Updating database. 2022/01/26 09:44:26 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:28 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:29 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:30 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:31 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:32 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:34 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:36 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:37 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:38 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:39 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:40 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:41 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:42 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:43 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:44 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:45 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:46 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:47 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:49 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:50 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:51 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:52 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:53 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:54 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:55 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:56 Failed to Ping DB, sleep for 1 second. 2022/01/26 09:44:56 Failed to connect DB after 30 seconds, time out.

seems that the signer.json secret cannot be written correctly:

"storage": { "backend": "postgres", "db_url": "postgres://harbor-postgres:wJor%3A%3Ay%7DGs15Y2%7DTw%3C37Y.%7DD@harbor-postgres-ha:5432/notarysigner?sslmode=disable", "default_alias": "defaultalias" }

Password is generated and should be wJor::y}Gs15Y2}Tw<37Y.}D

is there maybe an issue with special characters?

github-actions[bot] commented 2 years ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] commented 2 years ago

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

k9ne257 commented 9 months ago

Was this issue ever resolved? Am having a similar problem. I'd really appreciate it if you have any ideas on how to solve it.