Open so121183gak opened 2 years ago
I am seeing the same exact issue, We just deployed to minikube using the latest branch and install instructions For us each job exec is using ~10 gigs of space and I'm seeing tons of containers being retained.
Also for some reason the azure python package is in the awx-ee eating up over 500mb of space, this is kinda odd... I'm not sure why this would be included by default.
when a job runs in a pod, it uses a container based on the awx-ee image. This image is generated nightly and pushed to quay.io/ansible/awx-ee:latest, so if your execution environment has pull "always pull container before running", then your k8s cluster is going to pull that image day.
Also k8s has some garbage collection features that you may play around with to remove old images. https://kubernetes.io/docs/concepts/architecture/garbage-collection/
I would call this a duplicate of https://github.com/ansible/awx-operator/issues/881, and there is probably some background of complaints from the mailing list.
Please confirm the following
Summary
Hi everyone I used AWX for about 5 months, everything went well until an error popped up: Error creating pod: container failed to start, ImagePullBackOff
For some reason, I immediately thought that for some reason the disk space had run out and I was not mistaken. After analyzing what was happening, I calculated that most of the data was in the folder /var/lib/docker/volumes/minikube/_data/lib/docker/overlay2
I had to clean quay.io/ansible/awx-ee images in gcr.io/k8s-minikube/kicbase:v0.0.29 container.
Please tell me what kind of images they are, why they accumulate there and is it possible to automate the cleaning process using regular means?
AWX version
19.5.1
Select the relevant components
Installation method
minikube
Modifications
no
Ansible version
No response
Operating system
No response
Web browser
Firefox
Steps to reproduce
You can try to fill the disk space up to 99% and try to execute any playbook
Expected results
I did not know that the disk space was running out, I could not think that the AWX did not clear the cache with images
Actual results
While running the playbook, it gave me an error: Error creating pod: container failed to start, ImagePullBackOff
Then I realized that something was wrong and began to try to figure it out
Additional information
No response