search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
13.86k stars 3.4k forks source link

Migration not updating postgres tables correctly #10916

Open mikecurrancs opened 3 years ago

mikecurrancs commented 3 years ago

Please confirm the following

Summary

I have succesfully upgraded my AWX Install from 13.0.0 -> 17.0.1 and from 17.0.1 -> 17.1.0, however when I upgrade from 17.1.0 to 18.0.0 or 19.3.0 I encounter the same errors. I redirected all the output to a log, and I see the create tables fail, because the db already exists, and I dont see any alter statements that add the missing columns.

ESC[33mpostgres_1  |ESC[0m 2021-08-19 14:53:37.138 UTC [115] ERROR:  column "last_isolated_check" of relation "main_instance" does not exist at character 87
ESC[33mpostgres_1  |ESC[0m 2021-08-19 14:53:37.138 UTC [115] STATEMENT:  INSERT INTO "main_instance" ("uuid", "hostname", "ip_address", "created", "modified", "last_isolate
d_check", "version", "capacity", "capacity_adjustment", "enabled", "managed_by_policy", "cpu", "memory", "cpu_capacity", "mem_capacity") VALUES ('00000000-0000-0000-0000-00
0000000000', 'awx_1', NULL, '2021-08-19T14:53:37.138440+00:00'::timestamptz, '2021-08-19T14:53:37.138493+00:00'::timestamptz, NULL, '', 0, '1.00', true, true, 0, 0, 0, 0) R
ETURNING "main_instance"."id"
ESC[36mawx_1_1     |ESC[0m Traceback (most recent call last):
ESC[36mawx_1_1     |ESC[0m   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
ESC[36mawx_1_1     |ESC[0m     return self.cursor.execute(sql, params)
ESC[36mawx_1_1     |ESC[0m psycopg2.errors.UndefinedColumn: column "last_isolated_check" of relation "main_instance" does not exist
ESC[36mawx_1_1     |ESC[0m LINE 1: ... "hostname", "ip_address", "created", "modified", "last_isol...

ESC[33mpostgres_1  |ESC[0m 2021-08-19 14:53:39.171 UTC [116] ERROR:  column main_instancegroup.controller_id does not exist at character 129
ESC[33mpostgres_1  |ESC[0m 2021-08-19 14:53:39.171 UTC [116] STATEMENT:  SELECT "main_instancegroup"."id", "main_instancegroup"."name", "main_instancegroup"."created", "main_instancegroup"."modified", "main_instancegroup"."controller_id", "main_instancegroup"."is_container_group", "main_instancegroup"."credential_id", "main_instancegroup"."pod_spec_override", "main_instancegroup"."policy_instance_percentage", "main_instancegroup"."policy_instance_minimum", "main_instancegroup"."policy_instance_list" FROM "main_instancegroup" WHERE "main_instancegroup"."name" = 'tower'
ESC[36mawx_1_1     |ESC[0m Traceback (most recent call last):
ESC[36mawx_1_1     |ESC[0m   File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
ESC[36mawx_1_1     |ESC[0m     return self.cursor.execute(sql, params)
ESC[36mawx_1_1     |ESC[0m psycopg2.errors.UndefinedColumn: column main_instancegroup.controller_id does not exist
ESC[36mawx_1_1     |ESC[0m LINE 1: ...roup"."created", "main_instancegroup"."modified", "main_inst...
ESC[36mawx_1_1     |ESC[0m                                                              ^

Which causes and endless loop on the migration.

AWX version

17.1.0

Installation method

docker on linux

Modifications

no

Ansible version

2.9.6

Operating system

Ubuntu 20.04

Web browser

No response

Steps to reproduce

I am currently running 17.1.0 and I am following the steps at this link as directed in the README.md for version 18.0.0

Expected results

I expected the database to be migrated, and updated with any new columns between versions.

Actual results

When I look at the tables in question for postgres, I do not see any of the columns that are now expected.

Additional information

No response

mikecurrancs commented 3 years ago

I logged into tools_postgres_1 and ran the following alters, restarted the job make docker-compose and the errors went away for the migration effort.

alter table main_inventoryupdate add column source_script_id integer;

alter table main_instance add column last_isolated_check timestamp with time zone;

alter table main_instancegroup add column controller_id integer;

However, I got stuck then on awx_1 could not connect to awx:8013 afterwards, but I had no awx docker container running, just tools_awx_1 (created by the make docker-compose)

shanemcd commented 3 years ago

If you can get back into the broken state, I would be curious if there were any helpful errors in the operator pod.

mikecurrancs commented 3 years ago

@shanemcd Attaching the log file I created with the install so I can check the db queries vs the errors for missing columns newdeploy.log

mikecurrancs commented 3 years ago

And this is what it does after I've fixed those tables, and try to run docker-compose on an already migrated db

postgres_1  |
postgres_1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1  |
postgres_1  | 2021-08-20 20:30:05.705 UTC [1] LOG:  starting PostgreSQL 12.8 (Debian 12.8-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1  | 2021-08-20 20:30:05.705 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1  | 2021-08-20 20:30:05.705 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1  | 2021-08-20 20:30:05.706 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2021-08-20 20:30:05.720 UTC [26] LOG:  database system was shut down at 2021-08-20 20:27:18 UTC
postgres_1  | 2021-08-20 20:30:05.731 UTC [1] LOG:  database system is ready to accept connections
redis_1_1   | 1:C 20 Aug 2021 20:30:05.525 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1_1   | 1:C 20 Aug 2021 20:30:05.525 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1_1   | 1:C 20 Aug 2021 20:30:05.525 # Configuration loaded
redis_1_1   | 1:M 20 Aug 2021 20:30:05.525 * monotonic clock: POSIX clock_gettime
redis_1_1   | 1:M 20 Aug 2021 20:30:05.527 * Running mode=standalone, port=0.
redis_1_1   | 1:M 20 Aug 2021 20:30:05.527 # Server initialized
redis_1_1   | 1:M 20 Aug 2021 20:30:05.527 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1_1   | 1:M 20 Aug 2021 20:30:05.529 * Loading RDB produced by version 6.2.5
redis_1_1   | 1:M 20 Aug 2021 20:30:05.529 * RDB age 167 seconds
redis_1_1   | 1:M 20 Aug 2021 20:30:05.529 * RDB memory usage when created 0.79 Mb
redis_1_1   | 1:M 20 Aug 2021 20:30:05.529 * DB loaded from disk: 0.001 seconds
redis_1_1   | 1:M 20 Aug 2021 20:30:05.529 * The server is now ready to accept connections at /var/run/redis/redis.sock
awx_1_1     | [ -d "/awx_devel/awx.egg-info" ] || python3 /awx_devel/setup.py egg_info_dev
awx_1_1     | cp -f /tmp/awx.egg-link /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx.egg-link
awx_1_1     | mkdir -p /var/lib/awx/; \
awx_1_1     | if [ "/var/lib/awx/venv/" ]; then \
awx_1_1     |   . /var/lib/awx/venv//awx/bin/activate; \
awx_1_1     | fi; \
awx_1_1     | python -c "import awx; print(awx.__version__)" > /var/lib/awx/.awx_version; \
awx_1_1     |
awx_1_1     | if [ "/var/lib/awx/venv/" ]; then \
awx_1_1     |   . /var/lib/awx/venv//awx/bin/activate; \
awx_1_1     | fi; \
awx_1_1     | awx-manage migrate --noinput
awx_1_1     | Operations to perform:
awx_1_1     |   Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit
awx_1_1     | Running migrations:
awx_1_1     |   No migrations to apply.
awx_1_1     | if [ "/var/lib/awx/venv/" ]; then \
awx_1_1     |   . /var/lib/awx/venv//awx/bin/activate; \
awx_1_1     | fi; \
awx_1_1     | awx-manage provision_instance --hostname=awx_1; \
awx_1_1     | awx-manage register_queue --queuename=tower --instance_percent=100;\
awx_1_1     | if [ "tower" == "tower,thepentagon" ]; then \
awx_1_1     |   awx-manage provision_instance --hostname=isolated; \
awx_1_1     |   awx-manage register_queue --queuename='thepentagon' --hostnames=isolated --controller=tower; \
awx_1_1     |   awx-manage generate_isolated_key > /awx_devel/awx/main/isolated/authorized_keys; \
awx_1_1     | fi;
awx_1_1     | Instance already registered awx_1
awx_1_1     | Instance Group already registered tower
awx_1_1     | (changed: False)
awx_1_1     | /usr/local/lib/python3.6/site-packages/supervisor/options.py:475: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
awx_1_1     |   'Supervisord is running as root and it is searching '
awx_1_1     | 2021-08-20 20:30:20,009 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
awx_1_1     | Unlinking stale socket /var/run/supervisor/supervisor.sock
awx_1_1     | 2021-08-20 20:30:20,316 INFO RPC interface 'supervisor' initialized
awx_1_1     | 2021-08-20 20:30:20,317 CRIT Server 'unix_http_server' running without any HTTP authentication checking
awx_1_1     | 2021-08-20 20:30:20,317 INFO supervisord started with pid 164
awx_1_1     | 2021-08-20 20:30:21,320 INFO spawned: 'awx-dispatcher' with pid 167
awx_1_1     | 2021-08-20 20:30:21,322 INFO spawned: 'awx-receiver' with pid 168
awx_1_1     | 2021-08-20 20:30:21,323 INFO spawned: 'awx-uwsgi' with pid 169
awx_1_1     | 2021-08-20 20:30:21,333 INFO spawned: 'awx-daphne' with pid 175
awx_1_1     | 2021-08-20 20:30:21,337 INFO spawned: 'awx-nginx' with pid 176
awx_1_1     | 2021-08-20 20:30:21,341 INFO spawned: 'awx-wsbroadcast' with pid 177
awx_1_1     | 2021-08-20 20:30:21,344 INFO spawned: 'awx-rsyslogd' with pid 178
awx_1_1     | 2021-08-20 20:30:21,347 INFO spawned: 'awx-receptor' with pid 183
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | nginx -g "daemon off;"
awx_1_1     | nginx: [emerg] getpwnam("nginx") failed
awx_1_1     | make[1]: *** [Makefile:263: nginx] Error 1
awx_1_1     | make[1]: Leaving directory '/awx_devel'
awx_1_1     | 2021-08-20 20:30:21,457 INFO exited: awx-nginx (exit status 2; not expected)
awx_1_1     | INFO 2021/08/20 20:30:21 Running control service control
awx_1_1     | INFO 2021/08/20 20:30:22 Initialization complete
awx_1_1     | 2021-08-20 20:30:23,043 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:23,043 INFO success: awx-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:23,043 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:23,043 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:23,045 INFO spawned: 'awx-nginx' with pid 240
awx_1_1     | 2021-08-20 20:30:23,045 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:23,045 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:23,045 INFO success: awx-receptor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | nginx -g "daemon off;"
awx_1_1     | nginx: [emerg] getpwnam("nginx") failed
awx_1_1     | make[1]: *** [Makefile:263: nginx] Error 1
awx_1_1     | make[1]: Leaving directory '/awx_devel'
awx_1_1     | 2021-08-20 20:30:23,145 INFO exited: awx-nginx (exit status 2; not expected)
awx_1_1     | 2021-08-20 20:30:25,149 INFO spawned: 'awx-nginx' with pid 261
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | nginx -g "daemon off;"
awx_1_1     | nginx: [emerg] getpwnam("nginx") failed
awx_1_1     | make[1]: *** [Makefile:263: nginx] Error 1
awx_1_1     | make[1]: Leaving directory '/awx_devel'
awx_1_1     | 2021-08-20 20:30:25,275 INFO exited: awx-nginx (exit status 2; not expected)
awx_1_1     | 2021-08-20 20:30:25,485 INFO     [-] daphne.cli Starting server at tcp:port=8051:interface=127.0.0.1
awx_1_1     | 2021-08-20 20:30:25,485 INFO     Starting server at tcp:port=8051:interface=127.0.0.1
awx_1_1     | 2021-08-20 20:30:25,486 INFO     [-] daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
awx_1_1     | 2021-08-20 20:30:25,486 INFO     HTTP/2 support not enabled (install the http2 and tls Twisted extras)
awx_1_1     | 2021-08-20 20:30:25,487 INFO     [-] daphne.server Configuring endpoint tcp:port=8051:interface=127.0.0.1
awx_1_1     | 2021-08-20 20:30:25,487 INFO     Configuring endpoint tcp:port=8051:interface=127.0.0.1
awx_1_1     | 2021-08-20 20:30:25,491 INFO     [-] daphne.server Listening on TCP address 127.0.0.1:8051
awx_1_1     | 2021-08-20 20:30:25,491 INFO     Listening on TCP address 127.0.0.1:8051
awx_1_1     | *** Starting uWSGI 2.0.18 (64bit) on [Fri Aug 20 20:30:25 2021] ***
awx_1_1     | compiled with version: 8.4.1 20200928 (Red Hat 8.4.1-1) on 19 August 2021 20:33:19
awx_1_1     | os: Linux-5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021
awx_1_1     | nodename: awx_1
awx_1_1     | machine: x86_64
awx_1_1     | clock source: unix
awx_1_1     | detected number of CPU cores: 2
awx_1_1     | current working directory: /awx_devel
awx_1_1     | detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
awx_1_1     | !!! no internal routing support, rebuild with pcre support !!!
awx_1_1     | uWSGI running as root, you can use --uid/--gid/--chroot options
awx_1_1     | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
awx_1_1     | chdir() to /awx_devel/
awx_1_1     | your memory page size is 4096 bytes
awx_1_1     |  *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
awx_1_1     | detected max file descriptor number: 1048576
awx_1_1     | lock engine: pthread robust mutexes
awx_1_1     | thunder lock: disabled (you can enable it with --thunder-lock)
awx_1_1     | uwsgi socket 0 bound to TCP address 127.0.0.1:8050 fd 3
awx_1_1     | uWSGI running as root, you can use --uid/--gid/--chroot options
awx_1_1     | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
awx_1_1     | Python version: 3.6.8 (default, Mar 19 2021, 05:13:41)  [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)]
awx_1_1     | Set PythonHome to /var/lib/awx/venv/awx
awx_1_1     | Python main interpreter initialized at 0x1d6cbe0
awx_1_1     | uWSGI running as root, you can use --uid/--gid/--chroot options
awx_1_1     | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
awx_1_1     | python threads support enabled
awx_1_1     | your server socket listen backlog is limited to 100 connections
awx_1_1     | your mercy for graceful operations on workers is 60 seconds
awx_1_1     | mapped 609552 bytes (595 KB) for 5 cores
awx_1_1     | *** Operational MODE: preforking ***
awx_1_1     | uWSGI running as root, you can use --uid/--gid/--chroot options
awx_1_1     | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
awx_1_1     | *** uWSGI is running in multiple interpreter mode ***
awx_1_1     | spawned uWSGI master process (pid: 272)
awx_1_1     | spawned uWSGI worker 1 (pid: 273, cores: 1)
awx_1_1     | spawned uWSGI worker 2 (pid: 274, cores: 1)
awx_1_1     | spawned uWSGI worker 3 (pid: 275, cores: 1)
awx_1_1     | spawned uWSGI worker 4 (pid: 276, cores: 1)
awx_1_1     | spawned uWSGI worker 5 (pid: 277, cores: 1)
awx_1_1     | *** Stats server enabled on /tmp/stats.socket fd: 17 ***
awx_1_1     | Python auto-reloader enabled
awx_1_1     | 2021-08-20 20:30:26,606 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:283
awx_1_1     | 2021-08-20 20:30:26,641 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:284
awx_1_1     | 2021-08-20 20:30:26,646 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:287
awx_1_1     | 2021-08-20 20:30:26,667 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:288
awx_1_1     | 2021-08-20 20:30:26,761 WARNING  [-] awx.main.wsbroadcast Adding {'awx'} to websocket broadcast list
awx_1_1     | 2021-08-20 20:30:26,817 WARNING  [-] awx.main.wsbroadcast Connection from awx_1 to awx failed: 'Cannot connect to host awx:8013 ssl:False [Name or service not known]'.
awx_1_1     | 2021-08-20 20:30:26,952 WARNING  [-] awx.main.tasks Rejoining the cluster as instance awx_1.
awx_1_1     | 2021-08-20 20:30:27,478 INFO waiting for awx-rsyslogd to stop
awx_1_1     | 2021-08-20 20:30:27,479 INFO stopped: awx-rsyslogd (exit status 0)
awx_1_1     | 2021-08-20 20:30:27,483 INFO spawned: 'awx-rsyslogd' with pid 309
awx_1_1     | 2021-08-20 20:30:28,486 INFO spawned: 'awx-nginx' with pid 311
awx_1_1     | 2021-08-20 20:30:28,487 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | nginx -g "daemon off;"
awx_1_1     | nginx: [emerg] getpwnam("nginx") failed
awx_1_1     | make[1]: *** [Makefile:263: nginx] Error 1
awx_1_1     | make[1]: Leaving directory '/awx_devel'
awx_1_1     | 2021-08-20 20:30:28,629 INFO exited: awx-nginx (exit status 2; not expected)
awx_1_1     | 2021-08-20 20:30:28,629 INFO gave up: awx-nginx entered FATAL state, too many start retries too quickly
awx_1_1     | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x1d6cbe0 pid: 273 (default app)
awx_1_1     | running "exec: supervisorctl restart tower-processes:awx-dispatcher tower-processes:awx-receiver" (accepting1)...
awx_1_1     | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x1d6cbe0 pid: 275 (default app)
awx_1_1     | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x1d6cbe0 pid: 274 (default app)
awx_1_1     | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x1d6cbe0 pid: 276 (default app)
awx_1_1     | WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x1d6cbe0 pid: 277 (default app)
awx_1_1     | 2021-08-20 20:30:30,110 INFO waiting for awx-dispatcher to stop
awx_1_1     | 2021-08-20 20:30:30,118 INFO stopped: awx-dispatcher (terminated by SIGKILL)
awx_1_1     | 2021-08-20 20:30:30,119 INFO waiting for awx-receiver to stop
awx_1_1     | 2021-08-20 20:30:30,122 INFO stopped: awx-receiver (terminated by SIGKILL)
awx_1_1     | 2021-08-20 20:30:30,125 INFO spawned: 'awx-dispatcher' with pid 333
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | 2021-08-20 20:30:31,139 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:31,141 INFO spawned: 'awx-receiver' with pid 345
awx_1_1     | make[1]: Entering directory '/awx_devel'
awx_1_1     | 2021-08-20 20:30:31,826 WARNING  [-] awx.main.wsbroadcast Connection from awx_1 to awx failed: 'Cannot connect to host awx:8013 ssl:False [Name or service not known]'.
awx_1_1     | 2021-08-20 20:30:32,312 INFO success: awx-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | tower-processes:awx-dispatcher: stopped
awx_1_1     | tower-processes:awx-receiver: stopped
awx_1_1     | tower-processes:awx-dispatcher: started
awx_1_1     | tower-processes:awx-receiver: started
awx_1_1     | 2021-08-20 20:30:32,328 INFO waiting for awx-rsyslogd to stop
awx_1_1     | 2021-08-20 20:30:32,333 INFO stopped: awx-rsyslogd (exit status 0)
awx_1_1     | 2021-08-20 20:30:32,336 INFO spawned: 'awx-rsyslogd' with pid 368
awx_1_1     | 2021-08-20 20:30:33,052 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:372
awx_1_1     | 2021-08-20 20:30:33,056 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:373
awx_1_1     | 2021-08-20 20:30:33,060 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:374
awx_1_1     | 2021-08-20 20:30:33,064 WARNING  [-] awx.main.commands.run_callback_receiver scaling up worker pid:375
awx_1_1     | 2021-08-20 20:30:34,066 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
awx_1_1     | 2021-08-20 20:30:36,829 WARNING  [-] awx.main.wsbroadcast Connection from awx_1 to awx failed: 'Cannot connect to host awx:8013 ssl:False [Name or service not known]'.
awx_1_1     | 2021-08-20 20:30:41,833 WARNING  [-] awx.main.wsbroadcast Connection from awx_1 to awx failed: 'Cannot connect to host awx:8013 ssl:False [Name or service not known]'.
awx_1_1     | 2021-08-20 20:30:46,837 WARNING  [-] awx.main.wsbroadcast Connection from awx_1 to awx failed: 'Cannot connect to host awx:8013 ssl:False [Name or service not known]'.