d3al
commented
2 years ago Poking around a bit.
kubectl exec -ti deploy/awx -c awx-task -- /bin/bash
I look at the log /var/log/tower/dispatcher.log
I see two errors.
1.
2022-06-10 17:42:11,356 ERROR [9118a6b4cb024462860fa974127d7020] awx.main.dispatch.periodic PID:44 encountered an error while scheduling periodic tasks
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/periodic.py", line 37, in run
self.run_pending()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/schedule/__init__.py", line 94, in run_pending
self._run_job(job)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/schedule/__init__.py", line 147, in _run_job
ret = job.run()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/schedule/__init__.py", line 466, in run
ret = self.job_func()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/publish.py", line 86, in apply_async
with pg_bus_conn() as conn:
File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/__init__.py", line 50, in pg_bus_conn
conn = psycopg2.connect(dbname=conf['NAME'], host=conf['HOST'], user=conf['USER'], password=conf['PASSWORD'], port=conf['PORT'], **conf.get("OPTIONS", {}))
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg2/__init__.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name "awx-postgres" to address: Name or service not known
2022-06-10 17:52:03,407 WARNING [5bfc05f82254461185ccb17274b8a02d] awx.main.dispatch PID:143 job 67 (canceled) was canceled (rc=1)2.
2022-06-17 13:17:36,647 WARNING [5316cdd5244c4fbdb12adefbcb93a960] awx.main.dispatch PID:4942 job 222 (error) encountered an error (rc=None), please see task stdout for details.
2022-06-17 17:48:49,311 ERROR [763b1bebc6904b8ca3a2dd0578cbf79f] awx.main.dispatch PID:5742 Worker failed to run task awx.main.tasks.system.purge_old_stdout_files(*[], **{}
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/worker/task.py", line 90, in perform_work
result = self.run_callable(body)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/worker/task.py", line 65, in run_callable
return _call(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/system.py", line 326, in purge_old_stdout_files
for f in os.listdir(settings.JOBOUTPUT_ROOT):
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/awx/job_status'
2022-06-17 20:32:31,900 WARNING [20450ed92f3b4f119f8cfe66517489c4] awx.main.dispatch PID:5742 job 226 (canceled) was canceled (rc=1)I've checked /var/lib/awx/ and I don't see a job_status directory or file. I created a directory and ran the job again. I will check this log if the job gets stuck again, I have a feeling it's related to this folder not existing and the job fails. I will update soon.
r0b1ndot
Please confirm the following
Bug Summary
I did a clean install of awx (21.1.0) 2 weeks ago via operator on a digital ocean VM 4cpu, 8g ram.
Install went fine, created the templates, inventories manually from an awx install that is still in production (13). Sync project files via git and I'm able to run jobs. I'm hitting this particular problem that happens all the time, so much so I don't trust it can complete the job and may require multiple attempts.
I have many templates, but three of them are on a daily schedule and the issue doesn't happen on just these and it certainly happens if the job runs from schedule or manually. The jobs seem to randomly get stuck, might be on the same task, maybe on a different task. Maybe the job completes or maybe it gets stuck on a pass on the inventory. These jobs run 3 machines at a time, when I could have say 20 in the inventory. It's common that the first few passes there are no issues but it can get stuck on a machine on a task. It seems random but there is a good chance the job will get stuck.
If I let the job run, it's in a waiting state for several hours and eventually cancels itself. The task doesn't seem to be anything interesting. It seems the job might take longer to complete compared to the existing awx deployment. So I'm not sure if there is a timeout problem or if awx is waiting for a response but never comes. All of these machines are remote yet I've not had this problem on the v13 deployment.
I'm not sure what to look for so I'm just raising the issue to get direction.
AWX version
21.1.0
Select the relevant components
Installation method
kubernetes
Modifications
no
Ansible version
No response
Operating system
Ubuntu 22.04
Web browser
No response
Steps to reproduce
N/A
Expected results
Job completes without pausing.
Actual results
Job gets stuck on a random task but it's not consistent. Could be another task in the job or it may complete the job just fine.
Additional information
No response