hub_config.yml - Automation Hub returns 504

[joeashcraft]

I'm trying to use hub_config.yml to add 3 collection_remote's, "rh-certified", "validated", "community".
In my case, these 3 remotes already exist.

This task fails for me nearly every time because Automation Hub returns HTTP Error 504: Gateway Time-out. Automation Hub Task Manager shows no recent tasks.

AAP Job Output

...
TASK [infra.ah_configuration.collection_remote : Validating arguments against arg spec 'main']
ok: [localhost] ...

TASK [infra.ah_configuration.collection_remote : Add Automation Hub Collection Remote repository]
ok: [localhost] ...

TASK [infra.ah_configuration.collection_remote : Create Repository | Wait for the repository creation]
FAILED - RETRYING: [localhost]: Create Repository | Wait for the repository creation (15 retries left)
FAILED - RETRYING: [localhost]: Create Repository | Wait for the repository creation (14 retries left)
failed: [localhost] ... "msg": "Error while getting server version: The host sent back a server error: /api/galaxy/: HTTP Error 504: Gateway Time-out. ..."
...

Pulp logs (from journalctl)

...
Mar 29 15:41:01 dev-automationhub1  gunicorn[579418]: [2024-03-29 15:41:01 -0400] [579418] [CRITICAL] WORKER TIMEOUT (pid:579422)
...

Nginx Access Logs

==>/var/log/nginx/automationhub.access.log<==
10.1.2.3 - - [29/Mar/2024:15:27:05 -0400] "POST /api/galaxy/_ui/v1/auth/login/ HTTP/1.1" 504 160 https://dev-automationhub.example.com "Python-urllib/3.6" "-"
10.1.2.3 - admin [29/Mar/2024:15:28:00 -0400] "GET /api/galaxy/_ui/v1/me/ HTTP/1.1" 504 160 https://dev-automationhub.example.com "Python-urllib/3.6" "-"
10.1.2.3  - - [29/Mar/2024:15:37:01 -0400] "POST /api/galaxy/_ui/v1/auth/login/ HTTP/1.1" 504 160 https://dev-automationhub.example.com "Python-urllib/3.6" "-"
10.1.2.3  - admin [29/Mar/2024:15:38:06 -0400] "GET /api/galaxy/ HTTP/1.1" 504 160 https://dev-automationhub.example.com "Python-urllib/3.6" "-"
10.1.2.3  - admin [29/Mar/2024:15:39:01 -0400] "POST /api/galaxy/_ui/v1/auth/logout/ HTTP/1.1" 504 160 https://dev-automationhub.example.com "Python-urllib/3.6" "-"

Nginx Error Logs

==> /var/log/nginx/automationhub.error.log <==
2024/03/29 15:27:00 [error] 1722181#0: *109935 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.2.3, server: dev-automationhub1, request: "POST /api/galaxy/_ui/v1/auth/login/ HTTP/1.1", upstream: http://unix:/var/run/pulpcore-api/pulpcore-api.sock/api/galaxy/_ui/v1/auth/login/, host: "dev-automationhub.example.com", referrer: https://dev-automationhub.example.com
2024/03/29 15:28:00 [error] 1722176#0: *109943 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.2.3, server: dev-automationhub1, request: "GET /api/galaxy/_ui/v1/me/ HTTP/1.1", upstream: http://unix:/var/run/pulpcore-api/pulpcore-api.sock/api/galaxy/_ui/v1/me/, host: "dev-automationhub.example.com", referrer: https://dev-automationhub.example.com
2024/03/29 15:37:00 [error] 579325#0: *811 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.2.3, server: dev-automationhub1, request: "POST /api/galaxy/_ui/v1/auth/login/ HTTP/1.1", upstream: http://unix:/var/run/pulpcore-api/pulpcore-api.sock/api/galaxy/_ui/v1/auth/login/, host: "dev-automationhub.example.com", referrer: https://dev-automationhub.example.com
2024/03/29 15:38:01 [error] 579326#0: *821 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.2.3, server: dev-automationhub1, request: "GET /api/galaxy/ HTTP/1.1", upstream: http://unix:/var/run/pulpcore-api/pulpcore-api.sock/api/galaxy/, host: "dev-automationhub.example.com", referrer: https://dev-automationhub.example.com
2024/03/29 15:39:01 [error] 579326#0: *829 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.2.3, server: dev-automationhub1, request: "POST /api/galaxy/_ui/v1/auth/logout/ HTTP/1.1", upstream: http://unix:/var/run/pulpcore-api/pulpcore-api.sock/api/galaxy/_ui/v1/auth/logout/, host: "dev-automationhub.example.com", referrer: https://dev-automationhub.example.com

[djdanielsson]

this does not appear to be an issue with the template/playbook but with the module. I would create an issue https://github.com/ansible/galaxy_collection/issues/new/choose in this repo

[joeashcraft]

you're right! woops