Networking BROKE. "Socket path does not exist: /var/run/awx-receptor/receptor.sock"

fs30000 commented 1 month ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Networking does not work because receptor does not start.

AWX version

24.6.1

Select the relevant components

[ ] UI
[ ] UI (tech preview)
[X] API
[ ] Docs
[ ] Collection
[ ] CLI
[ ] Other

Installation method

docker development environment

Modifications

no

Ansible version

No response

Operating system

Rocky 9.4

Web browser

No response

Steps to reproduce

Do the usual procedure to install:

git clone -b 24.6.1 https://github.com/ansible/awx.git
cd awx/
make docker-compose-build
COMPOSE_UP_OPTS=-d make docker-compose
docker exec     tools_awx_1 make clean-ui ui-devel

# checking logs:
docker logs -f tools_awx_1

Expected results

2024-08-02 10:57:16,652 ERROR [-] awx.main.dispatch Encountered unhandled error in dispatcher main loop Traceback (most recent call last): File "/awx_devel/awx/main/dispatch/worker/base.py", line 237, in run self.worker.on_start() File "/awx_devel/awx/main/dispatch/worker/task.py", line 141, in on_start dispatch_startup() File "/awx_devel/awx/main/tasks/system.py", line 117, in dispatch_startup cluster_node_heartbeat() File "/awx_devel/awx/main/tasks/system.py", line 575, in cluster_node_heartbeat inspect_execution_and_hop_nodes(instance_list) File "/awx_devel/awx/main/tasks/system.py", line 514, in inspect_execution_and_hop_nodes mesh_status = ctl.simple_command('status') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/receptorctl/socket_interface.py", line 81, in simple_command self.connect() File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/receptorctl/socket_interface.py", line 99, in connect raise ValueError(f"Socket path does not exist: {path}") ValueError: Socket path does not exist: /var/run/awx-receptor/receptor.sock

Actual results

Networking does not work.

Additional information

With a fully updated Rocky 9.4, the build process was warning about setuptools not being recent enough for setuptools-scm 8.1 so i updated setuptools to 69.5.1.

Trying to start receptor manually fails. Even tough config is NOT empty.

docker exec -it  tools_awx_1    bash
bash-5.1# receptor --config /etc/receptor/receptor.conf
empty receptor config, skipping...
bash-5.1#
bash-5.1# cat /etc/receptor/receptor.conf
---
- node:
    id: awx-1
    firewallrules:
      - action: "reject"
        tonode: awx-1
        toservice: "control"

- log-level: info

- tcp-listener:
    port: 2222

#- tls-server:
#    name: mutual-tls
#    cert: /etc/receptor/certs/awx.crt
#    key: /etc/receptor/certs/awx.key
#    requireclientcert: true
#    clientcas: /etc/receptor/certs/ca.crt

- control-service:
    service: control
    filename: /var/run/awx-receptor/receptor.sock

- work-command:
    worktype: local
    command: ansible-runner
    params: worker
    allowruntimeparams: true
    verifysignature: False

- work-kubernetes:
    worktype: kubernetes-runtime-auth
    authmethod: runtime
    allowruntimeauth: true
    allowruntimepod: true
    allowruntimeparams: true
    verifysignature: False

- work-kubernetes:
    worktype: kubernetes-incluster-auth
    authmethod: incluster
    allowruntimeauth: true
    allowruntimepod: true
    allowruntimeparams: true
    verifysignature: False
bash-5.1#

blotto69 commented 1 month ago

Hello, Same problem on my side using the same deployment method as you on releases 24.6.0 and 24.6.1 on Rocky Linux 9.4 and 9.3. I was able to build and run AWX until a few days ago. Now, I get the same error message as you:

tools_awx_1 | 2024-08-03 00:11:47,925 INFO [-] awx.main.dispatch Running worker dispatcher listening to queues ['tower_broadcast_all', 'tower_settings_change', 'awx-1'] tools_awx_1 | 2024-08-03 00:11:47,926 INFO spawned: 'awx-receptor' with pid 621 tools_awx_1 | empty receptor config, skipping... tools_awx_1 | 2024-08-03 00:11:47,936 WARN exited: awx-receptor (exit status 0; not expected) tools_awx_1 | 2024-08-03 00:11:47,962 ERROR [-] awx.main.dispatch Encountered unhandled error in dispatcher main loop tools_awx_1 | Traceback (most recent call last): tools_awx_1 | File "/awx_devel/awx/main/dispatch/worker/base.py", line 237, in run tools_awx_1 | self.worker.on_start() tools_awx_1 | File "/awx_devel/awx/main/dispatch/worker/task.py", line 141, in on_start tools_awx_1 | dispatch_startup() tools_awx_1 | File "/awx_devel/awx/main/tasks/system.py", line 117, in dispatch_startup tools_awx_1 | cluster_node_heartbeat() tools_awx_1 | File "/awx_devel/awx/main/tasks/system.py", line 575, in cluster_node_heartbeat tools_awx_1 | inspect_execution_and_hop_nodes(instance_list) tools_awx_1 | File "/awx_devel/awx/main/tasks/system.py", line 514, in inspect_execution_and_hop_nodes tools_awx_1 | mesh_status = ctl.simple_command('status') tools_awx_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tools_awx_1 | File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/receptorctl/socket_interface.py", line 81, in simple_command tools_awx_1 | self.connect() tools_awx_1 | File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/receptorctl/socket_interface.py", line 99, in connect tools_awx_1 | raise ValueError(f"Socket path does not exist: {path}") tools_awx_1 | ValueError: Socket path does not exist: /var/run/awx-receptor/receptor.sock

Available if additional testing is needed

- name: Git checkout
  ansible.builtin.git:
    repo: "https://github.com/ansible/awx.git"
    dest: "{{ apps_dir }}/awx"
    version: 24.6.1
  become: false

- name: Run docker-compose-build
  community.general.make:
    chdir: "{{ apps_dir }}/awx"
    target: docker-compose-build
  become: false

- name: Run docker-compose -d
  community.general.make:
    chdir: "{{ apps_dir }}/awx"
    target: docker-compose
    params:
      COMPOSE_UP_OPTS: -d
  become: false

- name: Clean and build the UI
  community.docker.docker_container_exec:
    container: tools_awx_1
    command: make clean-ui ui-devel
  become: false

Update :

I just tried on CentOS 9 Stream and got the same result.

rabindangol commented 1 month ago

Not sure if this is related (I have older version) but I am getting similar error. https://github.com/ansible/awx/issues/15421

fs30000 commented 1 month ago

I also tried with Rocky 9.2. Same result.

fs30000 commented 1 month ago

Ok, managed to make it work with 23.9.0. Because with 24.6.1, this would happen: https://github.com/ansible/awx/issues/15404

Commands:

git clone -b 23.9.0 https://github.com/ansible/awx.git
cd awx/

export RECEPTOR_IMAGE=quay.io/ansible/receptor:v1.4.8   COMPOSE_UP_OPTS=-d
make docker-compose-build
make docker-compose

docker update --restart always tools_awx_1
docker update --restart always tools_postgres_1
docker update --restart always tools_redis_1

docker exec  tools_awx_1 make clean-ui ui-devel
docker exec  -it tools_awx_1  awx-manage changepassword admin

So far so good.

ansible / awx