search

ansible / awx-operator

An Ansible AWX operator for Kubernetes built with Operator SDK and Ansible. 🤖
https://www.github.com/ansible/awx
Apache License 2.0
1.26k stars 633 forks source link

no /var/lib/awx/projects #1429

Open linuxcube opened 1 year ago

linuxcube commented 1 year ago

Please confirm the following

Bug Summary

I am new to ansible awx, at least the kubernetes version of it....I used to test this project back in the older docker/compose days.

Today I have install a basic installation of the latest version as of this writing 04/29/2023 and upon login to the GUI wanted to created a new manual project. When I select the manual options I keep getting the following error:

"There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty, or all of the contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the "awx" system user, or have AWX directly retrieve your playbooks from source control using the Source Control Type option above."

AWX version

AWX 22.1.0

Select the relevant components

Installation method

kubernetes

Modifications

no

Ansible version

ansible [core 2.13.3]

Operating system

RHEL 9.1

Web browser

Chrome

Steps to reproduce

Stardard installation using kubernetes as per the docs. Below is a snapshot of my pods

[root@awx lib]# kubectl get pods NAME READY STATUS RESTARTS AGE awx-operator-controller-manager-67bcc886fb-cdhfm 2/2 Running 0 22h awx-task-769446cfd7-p65mq 4/4 Running 0 10h awx-web-5f5bc8fff7-kgvr7 3/3 Running 0 10h awx-postgres-13-0 1/1 Running 0 10h

Expected results

I expected the manual project to work without additional modifications. I tried creating the /var/lib/awx/projects folder on the host,task,web containers....they dont appear to make a difference when it came to project creation via the GUI. I even adjusted the permissions to be 777 and 755 for the /var/lib/awx/projects* and that does not appear to make a difference. The GUi keeps complaining about the lack of /var/lib/awx/projects.

Under my deployment yml definition i used the following parameters:

cat awx-instance-deployment.yml

apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx spec: service_type: nodeport projects_persistence: true projects_storage_access_mode: ReadWriteOnce web_extra_volume_mounts: |

Actual results

I continue to observed the following issue in the GUI and unable to create manual projects....I know the SCM is the ideal fashion, but I still would like to have the aiblity to use manual mode. PLease advise.

""There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty, or all of the contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the "awx" system user, or have AWX directly retrieve your playbooks from source control using the Source Control Type option above.""

Additional information

No response

fosterseth commented 1 year ago

this should fixed in this PR https://github.com/ansible/awx-operator/pull/1328

which landed in awx-operator 2.0.1, can you confirm that you are using operator 2.0.1?

veguzman commented 1 year ago

Hello Fosterseth

The version I have of awx-operator is 2.2.1 and the same problem still persists.

Please your help.

stuntguy3000 commented 1 year ago

Issue still persists with a fresh install, 2.5.3 & AWX 23.1.0 @fosterseth

kurokobo commented 1 year ago

I don't believe this issue still persists in AWX Opearator 2.0.1 or later. If you are on 2.0.1 or later and are still having this issue, it is probably a completely different issue caused by your configuration or your environment.

To troubleshoot this issue:

For usage question, ask on Matrix, or Ansible Community Forum:

Kaushal-fy commented 1 year ago

Reload helped

stuntguy3000 commented 11 months ago

I don't believe this issue still persists in AWX Opearator 2.0.1 or later. If you are on 2.0.1 or later and are still having this issue, it is probably a completely different issue caused by your configuration or your environment.

To troubleshoot this issue:

  • Ensure your playbooks are visible under project directory on /var/lib/awx/projects in awx-web pod.

    • kubectl -n <your_namespace> exec -it deployment/<your_awx>-web -c <your_awx>-web -- ls -l /var/lib/awx/projects/<your_project_directory>

    • Empty project directory is not available in UI

    • All playbooks have to be placed under project directory (means sub directory) on /var/lib/awx/projects, not directly under /var/lib/awx/projects

    • Accepted: /var/lib/awx/projects/my_project/site.yml, /var/lib/awx/projects/my_project/playbooks/site.yml

    • Not accepted: /var/lib/awx/projects/site.yml

  • Try super-reload (Shift + F5) on the "Create New Project" page. Outdated list of your project directories may be cached in your browser.

For usage question, ask on Matrix, or Ansible Community Forum:

Hi @kurokobo, I assure you this issue absolutely still exists in the very latest version of AWX Operator.

I have done a second fresh install and the issue persists. Based on the response of the users here, I am not the only one. I believe this is a permissions problem, but I don't fully understand your troubleshooting steps as they presume the existence of Projects and Runbooks.

bash-5.1$ ls -la /var/lib/awx/projects total 8 drwxrwxr-x 2 root 1000 4096 Dec 9 05:44 . drwxrwxr-x 1 root root 4096 Dec 9 05:46 .. bash-5.1$

What configuration option "in my environment", on a fresh install, causes this issue?

I'm deployed using k3s, how do I change the permissions on that folder as I am not root?

kurokobo commented 11 months ago

@stuntguy3000 What is the actual problem for you? Missing perms to create directories and files under /var/lib/awx/projects? Could you provide specs of your AWX? Which storageclass is used for your project PV?

stuntguy3000 commented 11 months ago

@kurokobo, appreciate the prompt reply.

For a bit of context, these instructions were followed to achieve the deployment.

Based on my understanding of the issue, inside my awx-web pod, the projects folder has the following permissions.

bash-5.1$ ls -la /var/lib/awx/projects
total 8
drwxrwxr-x 2 root 1000 4096 Dec 9 05:44 .
drwxrwxr-x 1 root root 4096 Dec 9 05:46 ..
bash-5.1$

The awx user inside the pod does not have RW access to the folder above. Further to that, I am unable to elevate using sudo or any other method, so I am unable to change the permissions.

I'm not totally sure if this answers your question, please let me know if I can provide more information:

root@awx:/home/admin# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                               STORAGECLASS   REASON   AGE
pvc-b4204791-e662-4ac4-8083-bed62c910e7b   8Gi        RWO            Delete           Bound    awx/postgres-13-awx-postgres-13-0   local-path              3d17h
pvc-ad8e0a65-ef0d-4bb7-9acf-1eb4c3bef850   5Gi        RWO            Delete           Bound    awx/static-data-pvc                 local-path              3d17h
pvc-cfc083e6-b16e-4a1c-805d-9dac11dc11c8   8Gi        RWO            Delete           Bound    awx/awx-projects-claim              local-path              3d17h
root@awx:/home/admin# kubectl describe pv
Name:              pvc-b4204791-e662-4ac4-8083-bed62c910e7b
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: rancher.io/local-path
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-path
Status:            Bound
Claim:             awx/postgres-13-awx-postgres-13-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          8Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [awx]
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/k3s/storage/pvc-b4204791-e662-4ac4-8083-bed62c910e7b_awx_postgres-13-awx-postgres-13-0
    HostPathType:  DirectoryOrCreate
Events:            <none>

Name:              pvc-ad8e0a65-ef0d-4bb7-9acf-1eb4c3bef850
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: rancher.io/local-path
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-path
Status:            Bound
Claim:             awx/static-data-pvc
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          5Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [awx]
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/k3s/storage/pvc-ad8e0a65-ef0d-4bb7-9acf-1eb4c3bef850_awx_static-data-pvc
    HostPathType:  DirectoryOrCreate
Events:            <none>

Name:              pvc-cfc083e6-b16e-4a1c-805d-9dac11dc11c8
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: rancher.io/local-path
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-path
Status:            Bound
Claim:             awx/awx-projects-claim
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          8Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [awx]
Message:
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/k3s/storage/pvc-cfc083e6-b16e-4a1c-805d-9dac11dc11c8_awx_awx-projects-claim
    HostPathType:  DirectoryOrCreate
Events:            <none>
kurokobo commented 11 months ago

@stuntguy3000 Thanks for updating, so, am I correct in saying that you are facing the issue of not being able to create a project directory in /var/lib/awx/projects in web pod in the first place, right?

Although a bit complicated, AWX that deployed with AWX Operator is implemented as following:

In summary for your case, no need to change permissions for /var/lib/awx/projects. Just try creating directories and files via task pod instead of web pod.

After that the same directories/files should be visible in web pod and should be usable as a Manual type project (if not, you can proceed to my troubelshooting steps).

As an alternative, since you seem to be using K3s and local-path storage class, creating directories and files in /var/lib/rancher/k3s/storage/pvc-cfc083e6-b16e-4a1c-805d-9dac11dc11c8_awx_awx-projects-claim (you can get this path from kubectl describe pv) on the K3s host would produce the same results.

emrehayta commented 11 months ago

@kurakobo

I have a similar Problem.

Could not find or access '/var/lib/awx/projects/ansible/serverconfigs/debian12/etc/aliases' on the Ansible Controller

From the pods task and web /var/lib/projects/ looks like: (And the file is also available)

sh-5.1$ ls -lha total 20K drwxrwxr-x. 4 awx 1000 4.0K Dec 14 11:10 . drwxrwxr-x. 1 root root 4.0K Dec 14 11:17 .. drwxr-xr-x. 9 awx root 4.0K Apr 12 2023 _8ansible -rwxr-xr-x. 1 awx root 0 Jul 27 2022 _8__ansible.lock lrwxrwxrwx. 1 awx root 11 Dec 14 11:10 ansible -> _8ansible drwxr-xr-x. 3 awx root 4.0K Nov 4 2022 .__awx_cache

Do you have any suggestions to solve this problem?

If i execute this playbook on my host without using AWX it works without any problems.

stuntguy3000 commented 11 months ago

@stuntguy3000 Thanks for updating, so, am I correct in saying that you are facing the issue of not being able to create a project directory in /var/lib/awx/projects in web pod in the first place, right?

Although a bit complicated, AWX that deployed with AWX Operator is implemented as following:

  • If projects_persistence: true is specified in AWX CR (I believe you did so), the same pv is mounted as /var/lib/awx/projects on both task and web pod
  • The permission for /var/lib/awx/projects is 775, and the owner is root:1000
  • The awx user in web pod is not a member of group 1000, but in task pod, awx user is a member of 1000 group
  • So /var/lib/awx/projects is read-only from web pod, but writable from task pod

In summary for your case, no need to change permissions for /var/lib/awx/projects. Just try creating directories and files via task pod instead of web pod.

After that the same directories/files should be visible in web pod and should be usable as a Manual type project (if not, you can proceed to my troubelshooting steps).

As an alternative, since you seem to be using K3s and local-path storage class, creating directories and files in /var/lib/rancher/k3s/storage/pvc-cfc083e6-b16e-4a1c-805d-9dac11dc11c8_awx_awx-projects-claim (you can get this path from kubectl describe pv) on the K3s host would produce the same results.

Appreciate the detailed reply - I've had success manually editing project files via a bash shell the web pod. Thank you!

kurokobo commented 11 months ago

@stuntguy3000 Congrats, so as I said this issue does not exist in the recent version of AWX Operator😃 Have fun!

stuntguy3000 commented 11 months ago

@stuntguy3000 Congrats, so as I said this issue does not exist in the recent version of AWX Operator😃 Have fun!

That's fair enough, I'll give you that. Given the complexity of the issue and that so many people keep running in to it, you can see how that conclusion can be drawn :)

kurokobo commented 11 months ago

@emrehayta Do you mean that you get that error in a task in your playbook? If so, your issue is a completely different one than this issue. Please create a topic in the forum, as it is not recommended to troubleshoot another issue here, and your issue is not a bug but usage question: https://forum.ansible.com/

emrehayta commented 11 months ago

ok thank you @kurokobo i created my question on this forum.

I wasn't aware whether this was a bug after all, because this playbook used to work and after a few updates to AWX it no longer worked.

AndreySysop commented 8 months ago

I confirm that there is a problem. I installed AWX 24.1.0 (awx-operator 2.14.0) on Debian. The path /var/lib/awx/projects was not created in the main OS. When creating manually, both before and after installing AWS, the WEB-console cursed when trying to create a manual project. I solved this problem locally for myself: in the awx-web container, I created the path /var/lib/awx/projects/blablablaprj and found its location in the OS (in my case - /var/lib/rancher/k3s/storage/pvc-3edf8d28-9a7b-497e-9b39-cab8800cad43_awx_static-data-pvc/) Then I created a symbolic link to it. Everything works, but I don't think this decision is correct.

dbrennand commented 5 months ago

@stuntguy3000 Thanks for updating, so, am I correct in saying that you are facing the issue of not being able to create a project directory in /var/lib/awx/projects in web pod in the first place, right?

Although a bit complicated, AWX that deployed with AWX Operator is implemented as following:

  • If projects_persistence: true is specified in AWX CR (I believe you did so), the same pv is mounted as /var/lib/awx/projects on both task and web pod
  • The permission for /var/lib/awx/projects is 775, and the owner is root:1000
  • The awx user in web pod is not a member of group 1000, but in task pod, awx user is a member of 1000 group
  • So /var/lib/awx/projects is read-only from web pod, but writable from task pod

In summary for your case, no need to change permissions for /var/lib/awx/projects. Just try creating directories and files via task pod instead of web pod.

After that the same directories/files should be visible in web pod and should be usable as a Manual type project (if not, you can proceed to my troubelshooting steps).

As an alternative, since you seem to be using K3s and local-path storage class, creating directories and files in /var/lib/rancher/k3s/storage/pvc-cfc083e6-b16e-4a1c-805d-9dac11dc11c8_awx_awx-projects-claim (you can get this path from kubectl describe pv) on the K3s host would produce the same results.

Hi @kurokobo

Thanks for this detailed breakdown here. It has helped me with some troubleshooting for a fresh deployment. I'm unable to login to AWX at all due to permission related issues in the awx-web pod:

image

kubectl -n awx logs awx-web-7c7df8c745-mstx8 awx-web
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/projects.py", line 67, in get_local_path_choices
    for x in os.listdir(settings.PROJECTS_ROOT)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects'
# AWX web
kubectl -n awx exec -it awx-web-7c7df8c745-mstx8 -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 37 Jul  1 15:32 .
drwxr-xr-x. 1 root root 17 Jun  4 19:40 ..
prw-------. 1 awx  root  0 Jul  1 15:32 awxfifo
drwxr-xr-x. 3 root root 19 Jun  4 19:40 .local
drwxrwsr-x. 2 root 1000  0 Jul  1 15:32 projects
drwxr-xr-x. 3 root root 20 Jun  4 19:40 public
drwxrwxr-x. 1 root root 40 Jun  4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun  4 19:36 venv
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ whoami
awx
bash-5.1$ cat /etc/group | grep '1000'

# vs AWX task
kubectl -n awx exec -it awx-task-5874fdf445-h5kc2 -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 22 Jul  1 15:32 .
drwxr-xr-x. 1 root root 17 Jun  4 19:40 ..
drwxr-xr-x. 3 root root 19 Jun  4 19:40 .local
drwxrwsr-x. 2 root 1000  0 Jul  1 15:32 projects
drwxr-xr-x. 3 root root 20 Jun  4 19:40 public
drwxrwxr-x. 1 root root 40 Jun  4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun  4 19:36 venv
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ cat /etc/group | grep '1000'

Do you know where the group 1000 is added to the awx user in awx-task?

I searched the upstream project and I could only see it being added in the entrypoint.sh script but that only happens when build_dev is true. So not applicable here for prod build of quay.io/ansible/awx:24.5.0.

I also noticed that the group 1000 (awx) doesn't exist in /etc/group in either the awx-web or awx-task.

Shouldn't the snippet from entrypoint.sh:

cat <<EOF >> /etc/group
awx:x:`id -u`:awx
EOF

Be added to launch_awx_task.sh and launch_awx_web.sh?

I may raise an issue upstream for this, as it seems weird for a group 1000 being used but doesn't exist in /etc/group?

kurokobo commented 5 months ago

https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074