AWXBackup: mkdir: cannot create directory '/backups/tower-openshift-backup-2024-04-17-141257': Permission denied

PWeverton commented 4 months ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Upgraded awx operator from 1.1.4 to 2.13.1 and started to get issues when trying to take backups. Here`s an example of the AWXBackup I have:

apiVersion: awx.ansible.com/v1beta1 kind: AWXBackup metadata: name: awx-demo namespace: awx-test spec: deployment_name: awx-demo backup_pvc: 'backup-pvc' no_log: false

Once applied, operator tries to create a folder for the backup on the db-management pod. However, its getting the issue permission denied

[backup : Set backup directory name] **\r\ntask path: /opt/ansible/roles/backup/tasks/postgres.yml:55\nok: [localhost] => {\"ansible_facts\": {\"backup_dir\": \"/backups/tower-openshift-backup-2024-04-17-141257\"}, \"changed\": false}\n\r\nTASK [backup : Create directory for backup] ****\r\ntask path: /opt/ansible/roles/backup/tasks/postgres.yml:59\nansible.cfg.\nfatal: [localhost]: FAILED! => {\"changed\": true, \"rc\": 1, \"return_code\": 1, \"stderr\": \"mkdir: cannot create directory '/backups/tower-openshift-backup-2024-04-17-141257': Permission denied\n\", \"stderr_lines\": [\"mkdir: cannot create directory '/backups/tower-openshift-backup-2024-04-17-141257': Permission denied\"]

AWX Operator version

2.13.1

AWX version

24.0.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

microk8s v1.28.8

Modifications

no

Steps to reproduce

Fresh installation and trying to create a backup using AWXBackup CR.

Expected results

Take the backup successfully

Actual results

Failed backup

Additional information

No response

Operator Logs

No response

jessicamack commented 4 months ago

Hello @PWeverton, can you read through this issue and see if it applies to your case https://github.com/ansible/awx-operator/issues/1775? The new postgres image is expecting to write to your dir as uid-26. There are some workarounds discussed to address the change.

PWeverton commented 4 months ago

Hello @jessicamack, thanks for replying. Well, the operator is having issues when trying to create the dir on db-management pod, so I don't think the issue you marked is related to it. However, I just reproduced the action items suggested there. Here's what I did:

Used operator 2.15.0, as that feature was added on this tag.
Then adjusted my AWX CR:

apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx-app spec: no_log: false service_type: nodeport postgres_data_volume_init: true postgres_init_container_commands: | chown 26:0 /var/lib/pgsql/data chmod 700 /var/lib/pgsql/data

Even after this change, the issue with the permissions still there.

kurokobo commented 4 months ago

This issue is not addressed by #1805 (postgres_data_volume_init and postgres_init_container_commands) since this issue is in following situation:

Occurs in ephemeral *-db-management pod instead of main PSQL pod
Occurs in backup pvc instead of main PSQL pvc
No init container for *-db-management pod is implemented in the current AWX Operator
From 2.13.0, the image for *-db-management pod has also been changed to sclorg's one

So we should implement init container for *-db-management pod and have a flag to modify owners/perms for backup pvc, or have a flag to run *-db-management pod as UID:0.

@rooftopcellist F.Y.I.

PWeverton commented 4 months ago

hi @kurokobo, any movement here? Thanks

ranvit commented 3 months ago

I just made PR #1854 , I'm able to take successful backups now if I run that init container once per PVC

pombaer commented 3 months ago

Please add this chang to the next Release since awxbackup also cannot create directory in my deployment because of permission issues. Changing the permissions on the NFS server to User ID 26 solved it but this is an manuall configuration step das workarround.

pombaer commented 3 months ago

May it helps someone, i workaround this problem by creating a cronjob which crates my backup and added an initcontainer which sets the permissions to 26:26 on the backup folder.

bar0n36 commented 3 months ago

I hit this issue after upgrading to 2.15.0. As per @pombaer first suggestion, I added another NFS mount and set the owner UID and GID to 26, then created a new PV/PVC and pushed the backup to that. For anyone using AWS EFS, you need to create an access point with the correct uid and gid and mount with that for it to work properly.

ansible / awx-operator