Control of free space on the hard disk on the AWX server

[so121183gak]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.

Summary

Hi everyone I used AWX for about 5 months, everything went well until an error popped up: Error creating pod: container failed to start, ImagePullBackOff

For some reason, I immediately thought that for some reason the disk space had run out and I was not mistaken. After analyzing what was happening, I calculated that most of the data was in the folder /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2

I had to clean quay.io/ansible/awx-ee images in gcr.io/k8s-minikube/kicbase:v0.0.29 container.

Please tell me what kind of images they are, why they accumulate there and is it possible to automate the cleaning process using regular means?

AWX version

19.5.1

Select the relevant components

• [ ] UI
• [x] API
• [x] Docs

Installation method

minikube

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

Firefox

Steps to reproduce

You can try to fill the disk space up to 99% and try to execute any playbook

Expected results

I did not know that the disk space was running out, I could not think that the AWX did not clear the cache with images

Actual results

While running the playbook, it gave me an error: Error creating pod: container failed to start, ImagePullBackOff

Then I realized that something was wrong and began to try to figure it out

Additional information

No response

[Vi-Robitaille]

I am seeing the same exact issue, We just deployed to minikube using the latest branch and install instructions For us each job exec is using ~10 gigs of space and I'm seeing tons of containers being retained.

Also for some reason the azure python package is in the awx-ee eating up over 500mb of space, this is kinda odd... I'm not sure why this would be included by default.

[fosterseth]

when a job runs in a pod, it uses a container based on the awx-ee image. This image is generated nightly and pushed to quay.io/ansible/awx-ee:latest, so if your execution environment has pull "always pull container before running", then your k8s cluster is going to pull that image day.

Also k8s has some garbage collection features that you may play around with to remove old images. https://kubernetes.io/docs/concepts/architecture/garbage-collection/

[AlanCoding]

I would call this a duplicate of https://github.com/ansible/awx-operator/issues/881, and there is probably some background of complaints from the mailing list.