AWX jobs are stuck in pending state

[prasanth-jaganathan]

Hello,

Triggered AWX jobs are remains still in the pending stage. And services related to AWX are in inactive state, even after restarting them.

[root@awx]# systemctl status awx-web.service awx-daphne.service awx-cbreceiver.service awx-dispatcher.service awx-wsbroadcast.service ● awx-web.service - AWX web service Loaded: loaded (/usr/lib/systemd/system/awx-web.service; disabled; vendor preset: disabled) Active: deactivating (stop-sigterm) since Fri 2021-01-15 14:02:53 UTC; 53s ago CGroup: /system.slice/awx-web.service ├─15504 /usr/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/tower/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 ├─15506 /usr/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/tower/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 ├─15507 /usr/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/tower/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 ├─15508 /usr/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/tower/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 ├─15509 /usr/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/tower/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 └─15510 /usr/bin/uwsgi --socket 127.0.0.1:8050 --module=awx.wsgi:application --logto=/var/log/tower/web.log --die-on-term --vacuum --uid awx --gid awx --processes=5 --harakiri=120 --no-orphans --master --max-requests=1000 --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768

Jan 15 14:02:50 sapansibleawx systemd[1]: Started AWX web service. Jan 15 14:02:53 sapansibleawx systemd[1]: Stopping AWX web service...

● awx-daphne.service - AWX daphne service Loaded: loaded (/usr/lib/systemd/system/awx-daphne.service; disabled; vendor preset: disabled) Active: inactive (dead) (Result: exit-code) since Fri 2021-01-15 14:02:53 UTC; 53s ago Process: 14173 ExecStart=/usr/bin/daphne -b 127.0.0.1 -p 8051 -b 127.0.0.1 -p 8051 --websocket_timeout -1 awx.asgi:channel_layer (code=exited, status=1/FAILURE) Main PID: 14173 (code=exited, status=1/FAILURE)

Jan 15 14:02:51 sapansibleawx systemd[1]: awx-daphne.service: main process exited, code=exited, status=1/FAILURE Jan 15 14:02:51 sapansibleawx systemd[1]: Unit awx-daphne.service entered failed state. Jan 15 14:02:51 sapansibleawx systemd[1]: awx-daphne.service failed. Jan 15 14:02:53 sapansibleawx systemd[1]: awx-daphne.service holdoff time over, scheduling restart. Jan 15 14:02:53 sapansibleawx systemd[1]: Stopped AWX daphne service.

● awx-cbreceiver.service - AWX cbreceiver service Loaded: loaded (/usr/lib/systemd/system/awx-cbreceiver.service; disabled; vendor preset: disabled) Active: inactive (dead) (Result: exit-code) since Fri 2021-01-15 14:02:54 UTC; 53s ago Process: 14170 ExecStart=/usr/bin/awx-manage run_callback_receiver (code=exited, status=1/FAILURE) Main PID: 14170 (code=exited, status=1/FAILURE)

Jan 15 14:02:52 sapansibleawx systemd[1]: awx-cbreceiver.service: main process exited, code=exited, status=1/FAILURE Jan 15 14:02:52 sapansibleawx systemd[1]: Unit awx-cbreceiver.service entered failed state. Jan 15 14:02:52 sapansibleawx systemd[1]: awx-cbreceiver.service failed. Jan 15 14:02:54 sapansibleawx systemd[1]: awx-cbreceiver.service holdoff time over, scheduling restart. Jan 15 14:02:54 sapansibleawx systemd[1]: Stopped AWX cbreceiver service.

● awx-dispatcher.service - AWX Dispatcher Loaded: loaded (/usr/lib/systemd/system/awx-dispatcher.service; disabled; vendor preset: disabled) Active: inactive (dead) (Result: exit-code) since Fri 2021-01-15 14:02:54 UTC; 53s ago Process: 14174 ExecStart=/usr/bin/awx-manage run_dispatcher (code=exited, status=1/FAILURE) Main PID: 14174 (code=exited, status=1/FAILURE)

Jan 15 14:02:52 sapansibleawx systemd[1]: awx-dispatcher.service: main process exited, code=exited, status=1/FAILURE Jan 15 14:02:52 sapansibleawx systemd[1]: Unit awx-dispatcher.service entered failed state. Jan 15 14:02:52 sapansibleawx systemd[1]: awx-dispatcher.service failed. Jan 15 14:02:54 sapansibleawx systemd[1]: awx-dispatcher.service holdoff time over, scheduling restart. Jan 15 14:02:54 sapansibleawx systemd[1]: Stopped AWX Dispatcher.

● awx-wsbroadcast.service - AWX channels worker service Loaded: loaded (/usr/lib/systemd/system/awx-wsbroadcast.service; disabled; vendor preset: disabled) Active: inactive (dead) (Result: exit-code) since Fri 2021-01-15 14:02:54 UTC; 53s ago Process: 14171 ExecStart=/usr/bin/awx-manage run_wsbroadcast (code=exited, status=1/FAILURE) Main PID: 14171 (code=exited, status=1/FAILURE)

Jan 15 14:02:52 sapansibleawx systemd[1]: awx-wsbroadcast.service: main process exited, code=exited, status=1/FAILURE Jan 15 14:02:52 sapansibleawx systemd[1]: Unit awx-wsbroadcast.service entered failed state. Jan 15 14:02:52 sapansibleawx systemd[1]: awx-wsbroadcast.service failed. Jan 15 14:02:54 sapansibleawx systemd[1]: awx-wsbroadcast.service holdoff time over, scheduling restart. Jan 15 14:02:54 sapansibleawx systemd[1]: Stopped AWX channels worker service.

########################### end #####################

Error log:- awx-manage

[root@awx tower]# /usr/bin/awx-manage run_wsbroadcast Traceback (most recent call last): File "/usr/bin/awx-manage", line 11, in sys.exit(manage()) File "/usr/lib/python3.6/site-packages/awx/init.py", line 136, in manage prepare_env() File "/usr/lib/python3.6/site-packages/awx/init.py", line 95, in prepare_env if not settings.DEBUG: # pragma: no cover File "/usr/lib/python3.6/site-packages/django/conf/init.py", line 79, in getattr self._setup(name) File "/usr/lib/python3.6/site-packages/django/conf/init.py", line 66, in _setup self._wrapped = Settings(settings_module) File "/usr/lib/python3.6/site-packages/django/conf/init.py", line 157, in init mod = importlib.import_module(self.SETTINGS_MODULE) File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 994, in _gcd_import File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "", line 678, in exec_module File "", line 219, in _call_with_frames_removed File "/usr/lib/python3.6/site-packages/awx/settings/production.py", line 17, in from .defaults import * # NOQA File "/usr/lib/python3.6/site-packages/awx/settings/defaults.py", line 46, in import ldap ModuleNotFoundError: No module named 'ldap'

Any solutions to this issue?

Thanks in advance.

[apoorvp2]

I am facing the same issue with AWX as the jobs get triggered and stays in Pending state . Kindly suggest for possible resolution

[kapilkhub88]

Hi,

Facing similar issue at AWX. Jobs keep going to pending state with no status updates. Getting difficult to know the execution status. Looking for the resolution steps.

[MrMEEE]

Project has been updated and moved...

Please move to the new version and report any issues

PROJECT MOVED: https://github.com/miracle-as/AWX-RPM

Issues at: https://github.com/miracle-as/AWX-RPM/issues

**Install guide, tools, utilities are located at: https://awx.wiki

LinkedIn group for Questions, support, talk and more: https://www.linkedin.com/groups/13694893/