khuedoan / homelab

Fully automated homelab from empty disk to running services with a single command.
https://homelab.khuedoan.com
GNU General Public License v3.0
8.21k stars 764 forks source link

Gitea Down Because The `configure-gitea` Run Failed #173

Open east4ming opened 1 week ago

east4ming commented 1 week ago

Describe the bug

After installing in high availability mode, gitea was rebooted, and after the reboot, gitea went down because the init container configure-gitea failed to run.

To reproduce

Steps to reproduce the behavior:

  1. Install Homelab
  2. Stop gitea
  3. See error log

Expected behavior

Please guide me step by step on how to troubleshoot this problem.

Logs and screenshots

init container configure-gitea logs: Command error: pq: cannot execute UPDATE in a read-only transaction

==== BEGIN GITEA CONFIGURATION ====
2024/11/05 07:47:40 cmd/migrate.go:33:runMigrate() [I] AppPath: /usr/local/bin/gitea
2024/11/05 07:47:40 cmd/migrate.go:34:runMigrate() [I] AppWorkPath: /data
2024/11/05 07:47:40 cmd/migrate.go:35:runMigrate() [I] Custom path: /data/gitea
2024/11/05 07:47:40 cmd/migrate.go:36:runMigrate() [I] Log path: /data/log
2024/11/05 07:47:40 cmd/migrate.go:37:runMigrate() [I] Configuration file: /data/gitea/conf/app.ini
2024/11/05 07:47:40 ...2@v2.25.7/command.go:267:Run() [I] PING DATABASE postgres
2024/11/05 07:47:40 ...2@v2.25.7/command.go:267:Run() [W] Table oauth2_application Column confidential_client db default is true, struct default is TRUE
2024/11/05 07:47:40 ...2@v2.25.7/command.go:267:Run() [W] Table system_setting Column version db default is , struct default is 1
2024/11/05 07:47:40 ...2@v2.25.7/command.go:267:Run() [W] Table user Column max_repo_creation db default is '-1', struct default is -1
2024/11/05 07:47:41 ...2@v2.25.7/command.go:267:Run() [W] Table label Column archived_unix db default is , struct default is NULL
Wait for redis to become avialable...
gitea-redis-cluster-headless.gitea.svc.cluster.local (10.0.0.21:6379) open
Admin account 'gitea_admin' already exist. Running update to sync password...
Command error: pq: cannot execute UPDATE in a read-only transaction
east4ming commented 1 week ago

The problem is resolved. Here's my workaround, for reference, since I don't know anything about postgres. So I didn't do any postgres sql related operations. Here's how it works.

  1. argocd disable auto-sync for argocd and gitea
  2. scale down gitea
  3. scale down gitea-postgresql-ha-pgpool
  4. scale down gitea-postgresql-ha-postgresql
  5. scale up gitea-postgresql-ha-postgresql replicas to 1
  6. scale up gitea-postgresql-ha-postgresql replicas to 2, then 3
  7. scale up gitea-postgresql-ha-postgresql
  8. scale up gitea
  9. RESOLVED.