AWX 13 broken due upstream issues

[ikke-t]

Permission issues:

• https://github.com/ansible/awx/issues/7545
• https://github.com/ansible/awx/issues/7544

So let's label awx_pod with 13.0 once those are fixed. This module is broken due those ATM at master branch. If you want to use a working module, select earlier version.

[Klaas-]

memcached was removed from awx in 12, I'd guess we need to change this role to reflect the upstream changes in kubernetes role; CCing @cfelder https://github.com/ansible/awx/pull/7240

[Klaas-]

https://github.com/ikke-t/awx_pod/blob/master/templates/settings.py.j2 https://github.com/ansible/awx/blob/devel/installer/roles/kubernetes/templates/configmap.yml.j2#L128-L202

and

https://github.com/ansible/awx/blob/devel/installer/roles/kubernetes/templates/deployment.yml.j2 https://github.com/ikke-t/awx_pod/blob/master/templates/awx.yml.j2

don't have my test setup ready at the moment so I won't get around to that soon :D

[ikke-t]

I suppose the order of getting this back to work would be to submit PRs to awx about those two starter script execute bits. Then waiting to get the upstream containers being rebuilt to work with PR changes. Once having those in place, there is place to make and test the above changes to drop memcached and update settings.

Creating the PR is small job, I wonder if @cfelder already has the changes in his repo?

I have a bit quieter time at work than usual, I could also do some of this one day.

[cfelder]

I am already on it (removing memcache).

I'll also post an example playbook here to run with a derived container image which has already fixed executable bits.

If I'll find the correct spot in the awx repo I can make a PR as well for fixing those Dockerfiles.

[cfelder]

The following example uses a derived container with fixed acls and removed preloading data into awx as well.

- name: run AWX on host
  hosts: all
  become: yes
  vars:
    awx_pod_name: awx
  tasks:
    - include_role:
        name: awx_pod
      vars:
        awx_admin_user: admin
        awx_admin_password: foobar
        awx_podman_dir: /tmp
        awx_host_port: 8052
        awx_pod_label: awx
        awx_awx_image: quay.io/cfelder/awx:nopreload-{{ awx_awx_version }}

Dockerfile

FROM docker.io/ansible/awx:13.0.0
LABEL maintainer="Christian Felder"

USER root

RUN dnf update -y && \
  dnf clean all -y
RUN sed -i '/.*awx-manage.*create_preload_data.*/d' \
  /usr/bin/launch_awx_task.sh \
  && \
 chmod 755 /usr/bin/launch_awx*.sh

USER 1000

needs: #19

[Klaas-]

If I recall right awx assumes it runs priviledged, so you won't need to change the runscripts but add priviledged to the web container. With 11 it seems this was optional for the web container, now that they are not seperated anymore it seems mandatory.

[Klaas-]

and for the no preload, I am guessing you could just configmap a launch_awx_task.sh into the image, then you don't need to rebuild it. But that would make a fine upstream change, create a env variable that you can set to skip it :)

[ikke-t]

please review if these changes fix the issues, I could just press the PR button: https://github.com/ansible/awx/commit/6c107dbd666c0166e7938745440e1363db4ab272

[cfelder]

If I recall right awx assumes it runs priviledged

RunAsUser and RunAsGroup are not honoured in podman versions shipped with EL7 and EL8 you would need at least podman 1.7

[Klaas-]

@cfelder https://github.com/ikke-t/awx_pod/blob/master/templates/awx.yml.j2#L216-L217 that also for the web container, that should help :)

[cfelder]

@cfelder https://github.com/ikke-t/awx_pod/blob/master/templates/awx.yml.j2#L216-L217 that also for the web container, that should help :)

I am still getting

$ podman logs -f awx_awxtask
/bin/sh: /usr/bin/launch_awx_task.sh: Permission denied

$ podman logs -f awx_awxweb
/bin/sh: /usr/bin/launch_awx.sh: Permission denied

although I added securityContext...

Looks like privileged: True is not honored in kube play

when using

$ podman run --privileged=true --rm -it docker.io/ansible/awx:13.0.0 /usr/bin/launch_awx.sh

the shell script is executed correctly.

Nevertheless shouldn't running unprivileged containers be preferred?

[Klaas-]

It should be honoured, otherwise the tasks container should have never been working :)

but upstream also changed the security context to unpriviledged for all containers: https://github.com/ansible/awx/commit/b13a175668e8d120046462f08f6a78b43ddc05cf

so maybe there is something else in the deployment that has changed (or the deployment really was never working for upstream 13)

[Koleon]

Hello guys, I'm very interested in the latest AWX running on podman (RHEL8). Are there any options how to help with it?

[brianmillett]

Not sure if this is progress, or chasing a red herring, but, looking at the generated /etc/containers/pods/awx.yaml, I added the following securityContext to the awx_awxweb container and the awx_awxtask container.

securityContext:
  allowPrivilegeEscalation: true
  capabilities: {}
  privileged: true
  readOnlyRootFilesystem: false
  runAsGroup: 0
  runAsUser: 0
  seLinuxOptions: {}

Running the podman play command from the systemd service, I get the awx to start, but the logs: podman logs -f awx-awx_awxweb & podman logs -f awx-awx_awxtask return the same errors: ModuleNotFoundError: No module named 'memcache'

Traceback (most recent call last): File "/usr/bin/awx-manage", line 8, in sys.exit(manage()) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py", line 154, in manage execute_from_command_line(sys.argv) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line utility.execute() File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv self.execute(*args, cmd_options) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 361, in execute self.check() File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 390, in check include_deployment_checks=include_deployment_checks, File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 377, in _run_checks return checks.run_checks(kwargs) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks new_errors = check(app_configs=app_configs) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique all_namespaces = _load_all_namespaces(resolver) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces url_patterns = getattr(resolver, 'url_patterns', []) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/functional.py", line 80, in get res = instance.dict[self.name] = self.func(instance) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/urls/resolvers.py", line 584, in url_patterns patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/functional.py", line 80, in get res = instance.dict[self.name] = self.func(instance) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/urls/resolvers.py", line 577, in urlconf_module return import_module(self.urlconf_name) File "/var/lib/awx/venv/awx/lib64/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "", line 678, in exec_module File "", line 219, in _call_with_frames_removed File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/urls.py", line 18, in url(r'^api/', include('awx.api.urls', namespace='api')), File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/urls/conf.py", line 34, in include urlconf_module = import_module(urlconf_module) File "/var/lib/awx/venv/awx/lib64/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "", line 678, in exec_module File "", line 219, in _call_with_frames_removed File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/api/urls/init.py", line 5, in from .urls import urlpatterns File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/api/urls/urls.py", line 74, in from .oauth2_root import urls as oauth2_root_urls File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/api/urls/oauth2_root.py", line 10, in from oauth2_provider import views File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/oauth2_provider/views/init.py", line 2, in from .base import AuthorizationView, TokenView, RevokeTokenView File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/oauth2_provider/views/base.py", line 62, in class AuthorizationView(BaseAuthorizationView, FormView): File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/oauth2_provider/views/base.py", line 85, in AuthorizationView server_class = oauth2_settings.OAUTH2_SERVER_CLASS File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py", line 85, in oauth2_getattribute val = settings.OAUTH2_PROVIDER.get(attr) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 500, in getattr_without_cache return getattr(self._wrapped, name) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 413, in getattr value = self._get_local(name) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 329, in _get_local self._preload_cache() File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 283, in _preload_cache if self.cache.get('_awx_conf_preload_expires', default=empty) is not empty: File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 137, in get value = self.cache.get(key, **kwargs) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/init.py", line 98, in getattr return getattr(caches[DEFAULT_CACHE_ALIAS], name) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/init.py", line 79, in getitem cache = _create_cache(alias) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/init.py", line 54, in _create_cache return backend_cls(location, params) File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 151, in init import memcache ModuleNotFoundError: No module named 'memcache'

[cfelder]

See #19 which removes memcache

[brianmillett]

See #19 which removes memcache

Thanks

I applied those changes, as well as the securityContext I mentioned earlier, and now I have a working podman awx 13.0.0

thanks.

[ikke-t]

Why does the web container need to be privileged? Sounds like there are some permission fixes to be done, normally any web server should not need any special privileges.

[ikke-t]

Or was that just to go around the fact that launchers were missing execute bits for the groups and others? That fix is btw now merged in to awx.

[brianmillett]

Or was that just to go around the fact that launchers were missing execute bits for the groups and others? That fix is btw now merged in to awx.

Yes, since only root could execute:

bash-4.4# ls -l /usr/bin/launch_awx.sh -rwx------. 1 root root 795 Jun 23 18:59 /usr/bin/launch_awx.sh

Good to know that it has been fixed.