Stanza backrestrepo Permission denied

[sorinpepelea]

I am unable to create path '/backrestrepo/hippo-backrest-shared-repo': [13] Permission denied in "stanza-create" pod. All other pods seem to run ok . So i tried everithing, changing FSGroup in configs, change runAsUser..nothing seems to work...pgbouncer, cluster and backrest pods runs ok. But stanza keeps complaining about insuficient rights.

StorageClass is hostpath Operator version 4.5.1

[sorinpepelea]

The PV and PVC are created for cluster, and also looks like there is no problem with write acces, except for stanza

[jkatz]

I would classify this as a bug:

https://github.com/CrunchyData/postgres-operator/blob/6069d0d6589097d21422b4e3f85a6622f91b4e9c/internal/operator/backrest/backup.go#L106

Does not appear to set any of the global security context settings on the pgBackRest jobs, so if you have the FSGroup disabled it is not respecting it. I'm curious why this has not come up sooner, though -- the pgBackRest repo does have these settings.

Potential workarounds in the interim include:

1. Running pgbackrest create-stanza directly on the PostgreSQL or pgBackRest repo Pod
2. Modifying the pgo-backrest-repo-template.json entry in the pgo-config ConfigMap and manually setting the SecurityContext

[jkatz]

Further looking into this:

• There are no persistent storage volumes mounted to pgBackRest "create stanza" job, so the security context in this case should not be needed.
• With hostpath, file ownership does need to be appropriately handled by the Kubernetes administrator (similar to what you probably did to ensure the PostgreSQL cluster could be written to). This would indicate that the issue could be with the writes on the pgBackRest repository itself, not the Job create-stanza Job.

I will further look at if we want to add an explicit SecurityContext to the pgBackRest Job Pod, but I'm now less convinced this is a bug and rather is a local environmental issue with the hostpath permissions.

Note that the pgBackRest repository requires the filesystem to be owned by a UID 2000. I would ensure that the pgBackRest repository Pod is able to do so.

[sorinpepelea]

thx for support. i've managed to make this work with local-storage class, and also with rook-ceph. problem is just wit microk8s-hostpath. i believe you're right - environmental issue with the hostpath permission . deploy creates the pv and pvc, but even with main cluster pod and also pgbackrest pod started, file creation in /barkrestrepo is denied. i tried manually to do what stanza does.but also fail since the problem lies with path persmission. Does not look like a bug, but it would be very nice if you could set pgbackrest pod to be able to work with hostpat storage. Another mention : also pgadmin has the same problem - acces denied .

[jkatz]

The PostgreSQL, pgBackRest repository, and pgAdmin 4 Pods all work with hostpath storage. I know this I tested this against my own hostpath storage environment :wink:

As mentioned, the pgBackRest Job that runs create-stanza amongst other things does not require any additional SecurityContext as it does not actually mount a writeable filesystem: that is handled by the other Pods.

The requires both ensuring that the permissions on your storage device are set to the correct ownership settings and that, if needed, you provide appropriate supplemental groups in your configuration.

Anyway, I don't believe there is any additional programmatic action needed.

[imranrazakhan]

@sorinpepelea How you were able to fix with local-storage / hostpath?

[academiaresf]

If anyone is searching for this, in my environment (microk8s, hostpath-storage, crunchy-pgo 5.6) the solution is removing the spec.backups.pgbackrest.global.repoN-path.