ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
14.05k stars 3.42k forks source link

Cannot run template job due to RuntimeError: Error creating pod: timed out waiting for the condition, ImagePullBackOff #11026

Open sc8608 opened 3 years ago

sc8608 commented 3 years ago

Please confirm the following

Summary

New to AWX. (it might not be a bug, but a proxy issue. However I have been investigated for a couple of days. And I have not seen other infomation on google indicating the same error message. Please help me.)

Successfully Installed with AWX operator 0.13.0 and minikube, behind coperate proxy. Successfully synced code from git. Encounted error when tempted running a testing template job. Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 1406, in run res = receptor_job.run() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 2935, in run return self._run_internal(receptor_ctl) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 3000, in _run_internal raise RuntimeError(detail) RuntimeError: Error creating pod: timed out waiting for the condition, ImagePullBackOff

AWX version

AWX 19.3.0

Installation method

minikube

Modifications

no

Ansible version

No response

Operating system

minikube on centos with docker driver

Web browser

Chrome

Steps to reproduce

created project, and synced playbooks from git repository. defined inventory and host with ssh username and password. defined template to run a simple playbook with command module. `- name: test connection hosts: all tasks:

        name: cmd-date
        command: date`

Encounted error when tempted running a testing template job. Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 1406, in run res = receptor_job.run() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 2935, in run return self._run_internal(receptor_ctl) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 3000, in _run_internal raise RuntimeError(detail) RuntimeError: Error creating pod: timed out waiting for the condition, ImagePullBackOff

Expected results

job executed successfully.

Actual results

The job stuck in running state for a few miniutes, then report the following error. Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 1406, in run res = receptor_job.run() File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 2935, in run return self._run_internal(receptor_ctl) File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/main/tasks.py", line 3000, in _run_internal raise RuntimeError(detail) RuntimeError: Error creating pod: timed out waiting for the condition, ImagePullBackOff

Additional information

Installed with AWX Operator 0.13.0, Minikube, around Sep 2021. The whole environment is behind coperate proxy, with ZScaler certificate. Minikube has been configured with ZScaler certificate. I also tried implement the cert on awx-demo-task.

my troubleshooting so far. I logged into awx-demo-task container, found that the proxy is not deployed by default(also i dont know what should be the documented way to pass the proxy setting down to awx-demo containers level). I tried adding proxy in awx/job setting/Extra Environment Variables, but still having the same error. { "GIT_SSL_NO_VERIFY": "True", "HTTP_PROXY": "http://[our proxy]:8080/", "HTTPS_PROXY": "http://[our proxy]:8080/", "NO_PROXY": "localhost,127.0.0.1,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24,192.168.49.0/24" }

chrismeyersfsu commented 3 years ago

@sc8608 It looks like the awx pod service account is unable to launch pods into the namespace that is in the configuration for the container group. By default it tries to launch into the same namespace that you are running in.

hambuergaer commented 3 years ago

I`m running into the same issue since Sep. 23rd without using a proxy. Any ideas what causes the problem?