tiredofit / docker-db-backup

Backup multiple database types on a scheduled basis with many customizable options
MIT License
866 stars 126 forks source link

[Kubernetes cronjob] pg_isready only works interactively (pod user permissions maybe?) #361

Open seano-vs opened 2 months ago

seano-vs commented 2 months ago

Summary

TL;DR: pg_isready only seems to work when executed interactively in the pod, as opposed to when the pod is executed. This happens after I had to manually add the PGSSLMODE=require env variable because it was throwing a /root/.postgresql/postgresql.crt: Permission denied error.

Steps to reproduce

What I did was I:

I suspect that this is a permissions issue with how the commands are being executed, but I'm not entirely sure.

I have the following k8s config:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-storage-backup
  namespace: mastodon
spec:
  schedule: "30 1 * * *"
  concurrencyPolicy: Forbid
  suspend: false
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          name: postgres-storage-backup
        spec:
          volumes:
            - name: postgres-completion
              configMap:
                name: postgres-completion
                defaultMode: 0500
          containers:
            - name: postgres-storage-backup
              image: tiredofit/db-backup:4.1.3
              imagePullPolicy: IfNotPresent
              command:
                - /init
                - backup-now
              volumeMounts:
                - name: postgres-completion
                  mountPath: "/script"
              env:
                - name: DEBUG_MODE
                  value: "TRUE"
                - name: PGSSLMODE
                  value: "require"
                - name: MODE
                  value: "MANUAL"
                - name: MANUAL_RUN_FOREVER
                  value: "FALSE"
                - name: CONTAINER_ENABLE_SCHEDULING
                  value: "FALSE"
                - name: CONTAINER_ENABLE_MONITORING
                  value: "FALSE"
                - name: DEFAULT_POST_SCRIPT
                  value: "/script/postgres.sh"
                - name: DEFAULT_BACKUP_LOCATION
                  value: 'S3'
                - name: DEFAULT_S3_BUCKET
                  valueFrom:
                    configMapKeyRef:
                      name: storage-backup
                      key: postgres_bucket
                - name: DEFAULT_S3_KEY_ID
                  valueFrom:
                    configMapKeyRef:
                      name: storage-backup
                      key: DEFAULT_S3_KEY_ID
                - name: DEFAULT_S3_KEY_SECRET
                  valueFrom:
                    configMapKeyRef:
                      name: storage-backup
                      key: DEFAULT_S3_KEY_SECRET
                - name: DEFAULT_S3_REGION
                  valueFrom:
                    configMapKeyRef:
                      name: storage-backup
                      key: DEFAULT_S3_REGION
                - name: DEFAULT_S3_HOST
                  valueFrom:
                    configMapKeyRef:
                      name: storage-backup
                      key: DEFAULT_S3_HOST
                - name: DB01_TYPE
                  value: "pgsql"
                - name: DB01_HOST
                  valueFrom:
                    configMapKeyRef:
                      name: mastodon-env-tf
                      key: DB_HOST
                - name: DB01_PORT
                  valueFrom:
                    configMapKeyRef:
                      name: mastodon-env-tf
                      key: DB_PORT
                - name: DB01_NAME 
                  valueFrom:
                    configMapKeyRef:
                      name: mastodon-env-tf
                      key: DB_NAME
                - name: DB01_USER
                  valueFrom:
                    configMapKeyRef:
                      name: mastodon-env-tf
                      key: DB_USER
                - name: DB01_PASS 
                  valueFrom:
                    configMapKeyRef:
                      name: mastodon-env-tf
                      key: DB_PASS
          restartPolicy: OnFailure
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1

What is the expected correct behavior?

pg_isready sees that the server is up and backs it up

Relevant logs and/or screenshots

I've attached the debug logs with everything sensitive scrubbed: private-logs.txt

Environment

Possible fixes

I've spent a fair amount of time debugging this, so I felt like there was just a point where it would be best to track my progress with a bug open

seano-vs commented 2 months ago

update: I see that the path listed in this part of the readme is incorrect, and it should be /etc/services.available/dbbackup-01/run instead of /etc/services.available/10-db-backup/run. At least, that's what is in my docker image.

It's now able to execute "run" as was mentioned in the docs, but it's still stuck at the pg_isready not working in an automated fashion.