"Failed to download remote objects and refs: error: cannot open .git/FETCH_HEAD: Permission denied when not running privileged pod

joyartoun commented 1 year ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hello all,

I am using this operator deployed via helm chart on OKD. I have noticed that if we are running without root privileges we are getting error "annot open .git/FETCH_HEAD: Permission denied when not running privileged pod" in the awx project update run.

This is the CR manifest

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  security_context_settings:
    runAsGroup: 0
    runAsUser: 0
    fsGroup: 0
    fsGroupChangePolicy: OnRootMismatch
  route_host: <redacted>
  create_preload_data: true
  route_tls_termination_mechanism: Edge
  garbage_collect_secrets: false
  ingress_type: route
  loadbalancer_port: 80
  image_pull_policy: IfNotPresent
  projects_storage_size: 20Gi
  projects_storage_access_mode: ReadWriteMany
  projects_persistence: true
  projects_storage_class: ceph-filesystem
  replicas: 1
  admin_user: admin
  loadbalancer_protocol: http
  nodeport_port: 30080
  task_privileged: true
  postgres_storage_requirements:
    requests:
      storage: 8Gi
    limits:
      storage: 50Gi
  postgres_storage_class: ceph-block

AWX Operator version

chart version 1.1.3

AWX version

AWX 21.10.2

Kubernetes platform

openshift

Kubernetes/Platform version

4.11

Modifications

no

Steps to reproduce

Install helm chart version 1.1.3 with the following manifest on openshift.

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
  namespace: awx
spec:
  route_host: <redacted>
  create_preload_data: true
  route_tls_termination_mechanism: Edge
  garbage_collect_secrets: false
  ingress_type: route
  loadbalancer_port: 80
  image_pull_policy: IfNotPresent
  projects_storage_size: 20Gi
  projects_storage_access_mode: ReadWriteMany
  projects_persistence: true
  projects_storage_class: ceph-filesystem
  replicas: 1
  admin_user: admin
  loadbalancer_protocol: http
  nodeport_port: 30080
  task_privileged: true
  postgres_storage_requirements:
    requests:
      storage: 8Gi
    limits:
      storage: 50Gi
  postgres_storage_class: ceph-block

Expected results

Being able to run without root privileges

Actual results

project update fails with error: cannot open .git/FETCH_HEAD: Permission denied

Additional information

No response

Operator Logs

No response

djyasin commented 1 year ago

Hello @joyartoun, We would like to gather a little bit more information from you.

Please let us know where the error message you provided came from specifically.

Please also shell into your container and run ls -la /var/lib/awx/projects/__$projectID/ in the respective project directory and provide us with that additional output as well.

joyartoun commented 1 year ago

@djyasin

The message "cannot open .git/FETCH_HEAD" came in the web gui when I tried to do a project update after I added a project in AWX. It was in the project update run logs.

Keep in mind that I currently run the awx pod in privileged mode, so the following is working. Do you want me to remove the privileged settings and also provide the output of that? let me know.

sh-5.1# ls -la /var/lib/awx/projects/_45__gitlab_ansible
total 8
drwxr-xr-x. 5 root root   15 Jan  5 09:53 .
drwxrwxrwx. 4 root root    3 Jan  5 10:28 ..
-rw-r--r--. 1 root root 1975 Jan  5 09:53 ansible.cfg
-rw-r--r--. 1 root root   71 Jan  5 09:53 fluentbit.yaml
drwxr-xr-x. 8 root root   13 Jan  5 10:37 .git
-rw-r--r--. 1 root root   30 Jan  5 09:53 .gitignore
drwxr-xr-x. 4 root root    4 Jan  5 09:53 inventory
-rw-r--r--. 1 root root   77 Jan  5 09:53 minio.yaml
-rw-r--r--. 1 root root  102 Jan  5 09:53 README.md
-rw-r--r--. 1 root root  229 Jan  5 09:53 requirements-python-2.6.txt
-rw-r--r--. 1 root root  908 Jan  5 09:53 requirements.txt
drwxr-xr-x. 7 root root    5 Jan  5 09:53 roles
-rw-r--r--. 1 root root   80 Jan  5 09:53 rsyslog.yml
-rw-r--r--. 1 root root   69 Jan  5 09:53 telegraf.yaml
-rw-r--r--. 1 root root  308 Jan  5 09:53 version.yaml
-rw-r--r--. 1 root root  116 Jan  5 09:53 win_connection_test.yml
-rw-r--r--. 1 root root   84 Jan  5 09:53 zabbix.yml

djyasin commented 1 year ago

Hello @joyartoun, Thank you for providing that additional information. We will need to do some more investigating around this on our end.

ansible / awx-operator