search

percona / percona-postgresql-operator

Percona Operator for PostgreSQL
https://www.percona.com/doc/kubernetes-operator-for-postgresql/index.html
Apache License 2.0
254 stars 50 forks source link

PgBackrest: unable to create stanza #744

Open henryx opened 2 months ago

henryx commented 2 months ago

Report

Pgbackrest cannot perform a backup to S3 because the endpoint address is generated in a wrong way

More about the problem

Reported logs is here:

2024-05-07T19:04:54.455Z    ERROR   unable to create stanza {"controller": "postgrescluster", "controllerGroup": "postgres-operator.crunchydata.com", "controllerKind": "PostgresCluster", "PostgresCluster": {"name":"fap-cluster-pg-db","namespace":"postgres"}, "namespace": "postgres", "name": "fap-cluster-pg-db", "reconcileID": "4b48a586-101e-492c-b832-12cd2eb6c17d", "reconciler": "pgBackRest", "error": "command terminated with exit code 49: ERROR: [049]: unable to get address for 'postgres.minio-api.domain.com': [-2] Name or service not known\n", "errorVerbose": "command terminated with exit code 49: ERROR: [049]: unable to get address for 'postgres.minio-api.domain.com': [-2] Name or service not known\n\ngithub.com/percona/percona-postgresql-operator/internal/pgbackrest.Executor.StanzaCreateOrUpgrade\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/pgbackrest/pgbackrest.go:96\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcileStanzaCreate\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:2650\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcilePGBackRest\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:1360\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/controller.go:356\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcileStanzaCreate\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:2657\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).reconcilePGBackRest\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/pgbackrest.go:1360\ngithub.com/percona/percona-postgresql-operator/internal/controller/postgrescluster.(*Reconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/internal/controller/postgrescluster/controller.go:356\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}

Steps to reproduce

  1. Create a secret with S3 credentials

    apiVersion: v1
kind: Secret
metadata:
name: a-cluster-pg-db-pgbackrest-secrets
type: Opaque
stringData:
s3.conf: |
[global]
repo1-s3-key=a_key
repo1-s3-key-secret=a_secret
repo1-storage-verify-tls=n
EOF

  2. Create a cluster instance via Helm specifying a custom S3 endpoint:

    backups:
pgbackrest:
configuration:
- secret:
    name: fap-cluster-pg-db-pgbackrest-secrets
repos:
- name: repo1
  schedules:
    full: "0 * * * *"
  s3:
    bucket: "postgres"
    endpoint: "https://minio-api.domain.com/"
    region: custom

Versions

  1. Kubernetes: 1.28.9+rke2r1
  2. Operator: 2.3.1
  3. Database: 16

Anything else?

According to the logs, it seems that S3 endpoint is constructed as {bucket}.{endpoint} instead of {endpoint}/{bucket}

discostur commented 1 month ago

@henryx the documentation says 

The repo2-s3-uri-style option [should be set to path ](https://pgbackrest.org/configuration.html#section-repository/option-repo-s3-uri-style) if you use S3-compatible storage (otherwise you might see “host not found error” in your backup job logs), and is not needed for Amazon S3.

https://docs.percona.com/percona-operator-for-postgresql/2.0/backups-storage.html

nick2432 commented 1 month ago

can i work on this?