Barman not being able to delete objects from Minio S3 when using with CloudNativePG

[vojtechmares]

Hello,

first, I have created an issue at CNPG repository (cloudnative-pg#4890), but most likely it is a Barman issue, not CNPG.

My issue is, that Barman is not able to delete objects from our Minio (S3) bucket.

With CNPG, we are using retention policy as said in documentation, unfortunately Barman is unable to do cleanup, causing backup storage to grow.

See the log:

{"level":"error","ts":"2024-06-11T01:00:07Z","logger":"barman","msg":"Error invoking barman-cloud-backup-delete","logging_pod":"harbor-db-1","options":["--endpoint-url","http://10.77.0.60:9000","--cloud-provider","aws-s3","--retention-policy","RECOVERY WINDOW OF 3 DAYS","s3://cybroslabs-backups/databases/harbor_v1/","harbor-db"],"stdout":"","stderr":"2024-06-11 01:00:07,604 [534758] ERROR: Barman cloud backup delete exception: An error occurred (XMinioInvalidObjectName) when calling the ListObjectsV2 operation: Object name contains unsupported characters.\n","error":"exit status 4","stacktrace":"github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/barman.DeleteBackupsByPolicy\n\tpkg/management/barman/backupdelete.go:89\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*BackupCommand).backupMaintenance\n\tpkg/management/postgres/backup.go:393\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*BackupCommand).run\n\tpkg/management/postgres/backup.go:291"}

I've checked the object names manually, but I do not see anything out of the ordinary.

My CNPG backup and policy configuration is as follows:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: converge43db
  namespace: cybroslabs-c43
spec:
  imageName: ghcr.io/cloudnative-pg/postgresql:15.7
  # ...
  # Postgres instance parameters
  postgresql:
    parameters:
      max_connections: "1000"
  # High Availability configuration
  minSyncReplicas: 1
  maxSyncReplicas: 1
  # Enable replication slots for HA in the cluster
  replicationSlots:
    highAvailability:
      enabled: true
  backup:
    retentionPolicy: "3d"
    barmanObjectStore:
      tags:
        backupRetentionPolicy: "expire"
      historyTags:
        backupRetentionPolicy: "keep"
      destinationPath: "s3://cybroslabs-backups/databases/converge43_v1"
      endpointURL: "http://10.77.0.60:9000"
      wal:
        compression: bzip2
      data:
        compression: bzip2
      s3Credentials:
        accessKeyId:
          name: cybroslabs-converge-backup-credentials
          key: s3AccessKey
        secretAccessKey:
          name: cybroslabs-converge-backup-credentials
          key: s3SecretKey

The Kubernetes Secret with Minio credentials exists.

[martinmarques]

Which version of barman-cloud are you using? I recall this was reported internally before and we fixed it. Particularly one fix in v3.7.0

[vojtechmares]

I am using the version bundled with CNPG.

I ran the following command inside Postgres container:

$ barman --version
3.10.0 Barman by EnterpriseDB (www.enterprisedb.com)

[vojtechmares]

🤦‍♂️ Barman Cloud...

$ barman-cloud-backup --version
barman-cloud-backup 3.9.0

cc @martinmarques

[vojtechmares]

I've fixed it.

There is a difference for the destination path... it can't contain a trailing slash, see:

- destinationPath: "s3://cybroslabs-backups/databases/converge43_v1/"
+ destinationPath: "s3://cybroslabs-backups/databases/converge43_v1"

That is a tiny change and used in many examples in CNPG documentation, but with Minio this does not work.