Execution Node not listening

satellite-no commented 1 year ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

Setting up a Execution node to work with AWX deployed using the AWX-operator It will not setup following the procedure outlined in the docs (https://github.com/ansible/awx/blob/devel/docs/execution_nodes.md#create-instance-in-awx).

I believe the issue is that its not listening on the tcp port it is only listening on the tcp6 port of 27199.

[rocky@ip-10-0-33-92 34.200.239.151_install_bundle]$ sudo netstat -plunt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1089/sshd: /usr/sbi
tcp6       0      0 :::27199                :::*                    LISTEN      17767/receptor
tcp6       0      0 :::22                   :::*                    LISTEN      1089/sshd: /usr/sbi
udp        0      0 127.0.0.1:323           0.0.0.0:*                           688/chronyd
udp6       0      0 ::1:323                 :::*                                688/chronyd

AWX version

21.11.0

Select the relevant components

[ ] UI
[ ] API
[ ] Docs
[ ] Collection
[ ] CLI
[X] Other

Installation method

kubernetes

Modifications

no

Ansible version

latest

Operating system

rocky

Web browser

No response

Steps to reproduce

Setup execution node using IP address. Following docs found here https://github.com/ansible/awx/blob/devel/docs/execution_nodes.md#create-instance-in-awx

(hosted in AWS) For testing allow 0.0.0.0/0 to 27199 in the Security Group.

Host has no Firewall install so no modifications needed.

Expected results

The AWX controller should be able to communicate with instance as soon as setup playbook is complete.

Actual results

Node is unavailable in controller. Also when I try to netcat to the port to validate it times out.

nc -v 34.200.239.151 27199                                   
nc: connectx to 34.200.239.151 port 27199 (tcp) failed: Operation timed out

Additional information

No response

ompals92 commented 1 year ago

Change execution environment image pull path or create new execution environment.

quay.io/ansible/awx-ee:21.11.0

satellite-no commented 1 year ago

@ompals92 thanks for the reply but I am using the latest execution environment on the operator instances. The execution node is a standalone non K8S Rocky Linux node and I don't believe it uses images for communication?

Still learning this aspect of AWX so forgive me.

satellite-no commented 1 year ago

Forgot this in my original post.. I'm also seeing the below log message in the AWX controllers pod container awx-demo-ee.

WARNING 2023/02/02 18:18:10 Backend connection failed (will retry): dial tcp 34.230.139.155:27199: i/o timeout

dillje01 commented 8 months ago

What is the solution for this?

BK-STAR77 commented 7 months ago

I have this error too. Any solution ?

kurokobo commented 7 months ago

Hi,

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
...
tcp6       0      0 :::27199                :::*                    LISTEN      17767/receptor

It's intended that netstat shows tcp6 only for Receptor. Usually this is a kind of "IPv4-mapped IPv6" so even if you only see tcp6, you should still be able to connect with IPv4.

So for your issue, ensure your firewall is open for Receptor. Not only for Sequrity Group, but also host-level firewalls such as firewalld.

sudo firewall-cmd --add-port=27199/tcp --permanent
sudo firewall-cmd --reload

I believe this is not a bug, so if you have any further questions, asking on the forum is preferred way.

BK-STAR77 commented 7 months ago

Hi,
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
...
tcp6       0      0 :::27199                :::*                    LISTEN      17767/receptor
It's intended that netstat shows tcp6 only for Receptor. Usually this is a kind of "IPv4-mapped IPv6" so even if you only see tcp6, you should still be able to connect with IPv4.

So for your issue, ensure your firewall is open for Receptor. Not only for Sequrity Group, but also host-level firewalls such as firewalld.
sudo firewall-cmd --add-port=27199/tcp --permanent
sudo firewall-cmd --reload
I believe this is not a bug, so if you have any further questions, asking on the forum is preferred way.

Thank you for your help but not the firewall. Strange...

systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@l203ekj001 /]# firewall-config
-bash: firewall-config: command not found
[root@l203ekj001 /]# firewall-cmd --list-all
FirewallD is not running

I will check the forum

ansible / awx