[Docs] Failed to set up AWX following the docker-compose installation guide

Andersson007 commented 1 year ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Trying to run AWX using https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md, stumbled upon:

Waiting for postgres to be ready to accept connections

I tried to use an external db. My inventory file contained (I also tried not to specify the db):
```
pg_host="my local laptop IP"
pg_password="awx"
pg_username="awx"
pg_database="awx"
```

The PG server was configured to accept connections from everywhere for the specified user which has superuser and login permissions. The firewall allows connection to postgres and selinux is disabled. I'm able to connect using the same creds with psql

$ psql awx -h SAME_LOCAL_IP -U awx -W
Password: 
psql (12.7)
Type "help" for help.

The output is (notice that it shows postgres version 12.15 but my external DB has version 12.7 which means it doesn't try to connect to my external DB:

postgres_1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1  | 
postgres_1  | 2023-06-22 09:05:53.918 UTC [1] LOG:  starting PostgreSQL 12.15 (Debian 12.15-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres_1  | 2023-06-22 09:05:53.918 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1  | 2023-06-22 09:05:53.918 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1  | 2023-06-22 09:05:53.921 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2023-06-22 09:05:53.940 UTC [26] LOG:  database system was interrupted; last known up at 2023-06-21 13:46:44 UTC
postgres_1  | 2023-06-22 09:05:54.052 UTC [26] LOG:  database system was not properly shut down; automatic recovery in progress
postgres_1  | 2023-06-22 09:05:54.053 UTC [26] LOG:  redo starts at 0/1652568
postgres_1  | 2023-06-22 09:05:54.053 UTC [26] LOG:  invalid record length at 0/16525A0: wanted 24, got 0
postgres_1  | 2023-06-22 09:05:54.053 UTC [26] LOG:  redo done at 0/1652568
postgres_1  | 2023-06-22 09:05:54.060 UTC [1] LOG:  database system is ready to accept connections
awx_1_1     | WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
awx_1_1     | [ -d "/awx_devel/awx.egg-info" ] || python3.9 /awx_devel/tools/scripts/egg_info_dev
awx_1_1     | mkdir -p /var/lib/awx/; \
awx_1_1     | if [ "/var/lib/awx/venv" ]; then \
awx_1_1     |   . /var/lib/awx/venv/awx/bin/activate; \
awx_1_1     | fi; \
awx_1_1     | python3.9 -c "import awx; print(awx.__version__)" > /var/lib/awx/.awx_version; \
awx_1_1     | 
awx_1_1     | Waiting for postgres to be ready to accept connections
awx_1_1     | Waiting for postgres to be ready to accept connections
... (repeats forever)

What am I doing wrong?

Then I ran docker system prune -a, commended the lines in the inventory (like it was by default) and tried to go through the guide again and it showed the same result.

AWX version

devel

Select the relevant components

[ ] UI
[ ] UI (tech preview)
[ ] API
[X] Docs
[ ] Collection
[ ] CLI
[ ] Other

Installation method

docker development environment

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Go through https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md. Specify DB connection params in https://github.com/ansible/awx/blob/devel/tools/docker-compose/inventory as explained in the guide (you need to setup postgres, create a user with login and permissions and allow connection with md5 for the user). Anyway, it does not work w/o the external DB. Frankly speaking, i created the DB after I failed to set it up w/o it.

Expected results

AWX works

Actual results

It doesn't work waiting for DB ans showing the below forever:

awx_1_1     | Waiting for postgres to be ready to accept connections

See the full related part of the log in the description

Additional information

No response

fosterseth commented 1 year ago

@Andersson007 I know the docs mention using an inventory file to connect to an external database, but that feature seems to be broken at the moment

I would focus on getting the dev environment up without the external database, as that most certainly should be working.

Can you paste the errors you see when trying to just run "make docker-compose"? Before running that, please delete your ghcr.io/ansible/awx_devel docker images on your system using docker rmi so that it pulls fresh images.

Andersson007 commented 1 year ago

@fosterseth hello, thanks for your response

Now on a completely fresh system from the just-updated devel branch i run:

make docker-compose-build 
make docker-compose

There are no errors but as the same as the last time it indefinitely prints awx_1_1 | Waiting for postgres to be ready to accept connections:

redis_1_1   | 1:M 26 Jun 2023 07:36:43.500 * The server is now ready to accept connections at /var/run/redis/redis.sock
postgres_1  | 2023-06-26 07:36:43.641 UTC [26] LOG:  redo starts at 0/16526C8
postgres_1  | 2023-06-26 07:36:43.641 UTC [26] LOG:  invalid record length at 0/1652700: wanted 24, got 0
postgres_1  | 2023-06-26 07:36:43.641 UTC [26] LOG:  redo done at 0/16526C8
postgres_1  | 2023-06-26 07:36:43.650 UTC [1] LOG:  database system is ready to accept connections
awx_1_1     | WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
awx_1_1     | [ -d "/awx_devel/awx.egg-info" ] || python3.9 /awx_devel/tools/scripts/egg_info_dev
awx_1_1     | running egg_info_dev
awx_1_1     | creating awx.egg-info
awx_1_1     | writing awx.egg-info/PKG-INFO
awx_1_1     | writing dependency_links to awx.egg-info/dependency_links.txt
awx_1_1     | writing entry points to awx.egg-info/entry_points.txt
awx_1_1     | writing top-level names to awx.egg-info/top_level.txt
awx_1_1     | mkdir -p /var/lib/awx/; \
awx_1_1     | if [ "/var/lib/awx/venv" ]; then \
awx_1_1     |   . /var/lib/awx/venv/awx/bin/activate; \
awx_1_1     | fi; \
awx_1_1     | python3.9 -c "import awx; print(awx.__version__)" > /var/lib/awx/.awx_version; \
awx_1_1     | 
awx_1_1     | Waiting for postgres to be ready to accept connections
awx_1_1     | Waiting for postgres to be ready to accept connections
awx_1_1     | Waiting for postgres to be ready to accept connections
awx_1_1     | Waiting for postgres to be ready to accept connections
awx_1_1     | Waiting for postgres to be ready to accept connections
...

In another cli tab i tried to clean and build the ui docker exec tools_awx_1 make clean-ui ui-devel and then create an admin user as explained in the guide, and, as expected, it failed with a DB connection error:

$ docker exec -ti tools_awx_1 awx-manage createsuperuser

2023-06-26 07:43:29,986 WARNING  [-] awx.conf.settings Database settings are not available, using defaults. error: connection failed: Success
System check identified some issues:

WARNINGS:
?: (staticfiles.W004) The directory '/awx_devel/awx/ui_next/build' in the STATICFILES_DIRS setting does not exist.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
    self.connect()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 269, in get_new_connection
    connection = self.Database.connect(**conn_params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
    raise ex.with_traceback(None)
psycopg.OperationalError: connection failed: No route to host
    Is the server running on host "postgres" (172.18.0.3) and accepting
    TCP/IP connections on port 5432?

fosterseth commented 1 year ago

@Andersson007 postgres_1 | 2023-06-26 07:36:43.650 UTC [1] LOG: database system is ready to accept connections

seems your postgres container is up, but something is prevent tools_awx_1 from accessing that service.

what I would try:

delete your docker containers
delete your docker volumes
delete docker networks (specifically the one associated with awx, on my machine is called _sources_default but might be different on yours)
- you can use docker network ls to get a list of networks

Then try running make docker-compose again. Does that help?

Andersson007 commented 1 year ago

i'm right back from PTO now and I'll test the proposed solution this week, thanks

Andersson007 commented 1 year ago

@fosterseth it helped, thanks!

I caught another error addressed in https://github.com/ansible/awx/pull/14224. So closing this issue as solved. Thanks!

ansible / awx