search

ansible / awx

AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Other
14.02k stars 3.42k forks source link

AWX Receptor Node: IPv6 flapping (working, broken) #15271

Closed discostur closed 4 months ago

discostur commented 4 months ago

Please confirm the following

Bug Summary

I deployed AWX (v24.5.0) in k8s via helm. I added a receptor node (execution). If the server which i want to provision has an ipv4 address everything is working normal.

If the server i want to run my playbooks against as an ipv6 address, sometimes the job finishes successful and sometimes it failes. It is exactly the same job template. If i just click on re-run it sometimes works and sometimes not. Cannot see any pattern when it works and when not ...

AWX version

24.5.0

Select the relevant components

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

Debian

Web browser

Firefox

Steps to reproduce

Expected results

Working

Identity added: /runner/artifacts/1145/ssh_key_data (/runner/artifacts/1145/ssh_key_data)
ansible-playbook [core 2.15.12]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /runner/artifacts/1145/callback/awx_display.py
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: CloudCheckSSH.yml ****************************************************
Positional arguments: CloudCheckSSH/CloudCheckSSH.yml
verbosity: 4
remote_user: shell
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/runner/inventory',)
subset: 2001:4178:6:1416:0000:000b:a:14
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in CloudCheckSSH/CloudCheckSSH.yml

PLAY [--> CloudCheckSSH] *******************************************************

TASK [Gathering Facts] *********************************************************
task path: /runner/project/CloudCheckSSH/CloudCheckSSH.yml:4
<2001:4178:6:1416:0000:000b:a:14> ESTABLISH SSH CONNECTION FOR USER: root
<2001:4178:6:1416:0000:000b:a:14> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/d04603209b"' 2001:4178:6:1416:0000:000b:a:14 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<2001:4178:6:1416:0000:000b:a:14> (0, b'/root\\n', b'OpenSSH_8.7p1, OpenSSL 3.2.1 30 Jan 2024\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0000:000b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \\'final\\'\\r\\ndebug2: match not found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug1: configuration requests final Match pass\\r\\ndebug2: resolve_canonicalize: hostname 2001:4178:6:1416:0000:000b:a:14 is address\\r\\ndebug2: resolve_canonicalize: canonicalised address "2001:4178:6:1416:0000:000b:a:14" => "2001:4178:6:1416:0:b:a:14"\\r\\ndebug1: re-parsing configuration\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0:b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \\'final\\'\\r\\ndebug2: match found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts\\' -> \\'/root/.ssh/known_hosts\\'\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts2\\' -> \\'/root/.ssh/known_hosts2\\'\\r\\ndebug1: auto-mux: Trying existing master\\r\\ndebug1: Control socket "/runner/cp/d04603209b" does not exist\\r\\ndebug3: ssh_connect_direct: entering\\r\\ndebug1: Connecting to 2001:4178:6:1416:0:b:a:14 [2001:4178:6:1416:0:b:a:14] port 22.\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: fd 3 clearing O_NONBLOCK\\r\\ndebug1: Connection established.\\r\\ndebug3: timeout: 9997 ms remain after connect\\r\\ndebug1: identity file /root/.ssh/id_rsa type -1\\r\\ndebug1: identity file /root/.ssh/id_rsa-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_dsa type -1\\r\\ndebug1: identity file /root/.ssh/id_dsa-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa_sk type -1\\r\\ndebug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519 type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519_sk type -1\\r\\ndebug1: identity file /root/.ssh/id_ed25519_sk-cert type -1\\r\\ndebug1: identity file /root/.ssh/id_xmss type -1\\r\\ndebug1: identity file /root/.ssh/id_xmss-cert type -1\\r\\ndebug1: Local version string SSH-2.0-OpenSSH_8.7\\r\\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u2\\r\\ndebug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u2 pat OpenSSH* compat 0x04000000\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: Authenticating to 2001:4178:6:1416:0:b:a:14:22 as \\'root\\'\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory\\r\\ndebug3: order_hostkeyalgs: no algorithms matched; accept original\\r\\ndebug3: send packet: type 20\\r\\ndebug1: SSH2_MSG_KEXINIT sent\\r\\ndebug3: receive packet: type 20\\r\\ndebug1: SSH2_MSG_KEXINIT received\\r\\ndebug2: local client KEXINIT proposal\\r\\ndebug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,kex-strict-c-v00@openssh.com\\r\\ndebug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256\\r\\ndebug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr\\r\\ndebug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes128-gcm@openssh.com,aes128-ctr\\r\\ndebug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512\\r\\ndebug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512\\r\\ndebug2: compression ctos: zlib@openssh.com,zlib,none\\r\\ndebug2: compression stoc: zlib@openssh.com,zlib,none\\r\\ndebug2: languages ctos: \\r\\ndebug2: languages stoc: \\r\\ndebug2: first_kex_follows 0 \\r\\ndebug2: reserved 0 \\r\\ndebug2: peer server KEXINIT proposal\\r\\ndebug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com\\r\\ndebug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519\\r\\ndebug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\\r\\ndebug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\\r\\ndebug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\\r\\ndebug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\\r\\ndebug2: compression ctos: none,zlib@openssh.com\\r\\ndebug2: compression stoc: none,zlib@openssh.com\\r\\ndebug2: languages ctos: \\r\\ndebug2: languages stoc: \\r\\ndebug2: first_kex_follows 0 \\r\\ndebug2: reserved 0 \\r\\ndebug3: kex_choose_conf: will use strict KEX ordering\\r\\ndebug1: kex: algorithm: curve25519-sha256\\r\\ndebug1: kex: host key algorithm: ssh-ed25519\\r\\ndebug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: zlib@openssh.com\\r\\ndebug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: zlib@openssh.com\\r\\ndebug1: kex: curve25519-sha256 need=32 dh_need=32\\r\\ndebug1: kex: curve25519-sha256 need=32 dh_need=32\\r\\ndebug3: send packet: type 30\\r\\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\\r\\ndebug3: receive packet: type 31\\r\\ndebug1: SSH2_MSG_KEX_ECDH_REPLY received\\r\\ndebug1: Server host key: ssh-ed25519 SHA256:atqPBuFfxO6Sc9gPTvfkVqgaxu03/rGssvmMhSUg00U\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory\\r\\ndebug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory\\r\\nWarning: Permanently added \\'2001:4178:6:1416:0:b:a:14\\' (ED25519) to the list of known hosts.\\r\\ndebug1: check_host_key: hostkey not known or explicitly trusted: disabling UpdateHostkeys\\r\\ndebug3: send packet: type 21\\r\\ndebug1: ssh_packet_send2_wrapped: resetting send seqnr 3\\r\\ndebug2: set_newkeys: mode 1\\r\\ndebug1: rekey out after 4294967296 blocks\\r\\ndebug1: SSH2_MSG_NEWKEYS sent\\r\\ndebug1: expecting SSH2_MSG_NEWKEYS\\r\\ndebug3: receive packet: type 21\\r\\ndebug1: ssh_packet_read_poll2: resetting read seqnr 3\\r\\ndebug1: SSH2_MSG_NEWKEYS received\\r\\ndebug2: set_newkeys: mode 0\\r\\ndebug1: rekey in after 4294967296 blocks\\r\\ndebug1: Will attempt key: /runner/artifacts/1145/ssh_key_data RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI agent\\r\\ndebug1: Will attempt key: /root/.ssh/id_rsa \\r\\ndebug1: Will attempt key: /root/.ssh/id_dsa \\r\\ndebug1: Will attempt key: /root/.ssh/id_ecdsa \\r\\ndebug1: Will attempt key: /root/.ssh/id_ecdsa_sk \\r\\ndebug1: Will attempt key: /root/.ssh/id_ed25519 \\r\\ndebug1: Will attempt key: /root/.ssh/id_ed25519_sk \\r\\ndebug1: Will attempt key: /root/.ssh/id_xmss \\r\\ndebug2: pubkey_prepare: done\\r\\ndebug3: send packet: type 5\\r\\ndebug3: receive packet: type 7\\r\\ndebug1: SSH2_MSG_EXT_INFO received\\r\\ndebug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>\\r\\ndebug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)\\r\\ndebug3: receive packet: type 6\\r\\ndebug2: service_accept: ssh-userauth\\r\\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\\r\\ndebug3: send packet: type 50\\r\\ndebug3: receive packet: type 51\\r\\ndebug1: Authentications that can continue: publickey,password\\r\\ndebug3: start over, passed a different list publickey,password\\r\\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\\r\\ndebug3: authmethod_lookup publickey\\r\\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\\r\\ndebug3: authmethod_is_enabled publickey\\r\\ndebug1: Next authentication method: publickey\\r\\ndebug1: Offering public key: /runner/artifacts/1145/ssh_key_data RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI agent\\r\\ndebug3: send packet: type 50\\r\\ndebug2: we sent a publickey packet, wait for reply\\r\\ndebug3: receive packet: type 60\\r\\ndebug1: Server accepts key: /runner/artifacts/1145/ssh_key_data RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI agent\\r\\ndebug3: sign_and_send_pubkey: RSA SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI\\r\\ndebug3: sign_and_send_pubkey: signing using rsa-sha2-256 SHA256:+mnZxEhj4lNVhNfzvE860S9Yz7I5nMkYNBJe8HaGOnI\\r\\ndebug3: send packet: type 50\\r\\ndebug3: receive packet: type 52\\r\\ndebug1: Enabling compression at level 6.\\r\\nAuthenticated to 2001:4178:6:1416:0:b:a:14 ([2001:4178:6:1416:0:b:a:14]:22) using "publickey".\\r\\ndebug1: pkcs11_del_provider: called, provider_id = (null)\\r\\ndebug1: setting up multiplex master socket\\r\\ndebug3: muxserver_listen: temporary control path /runner/cp/d04603209b.Q9xNMF20HvKiTMbY\\r\\ndebug2: fd 4 setting O_NONBLOCK\\r\\ndebug3: fd 4 is O_NONBLOCK\\r\\ndebug3: fd 4 is O_NONBLOCK\\r\\ndebug1: channel 0: new [/runner/cp/d04603209b]\\r\\ndebug3: muxserver_listen: mux listener channel 0 fd 4\\r\\ndebug2: fd 3 setting TCP_NODELAY\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x20\\r\\ndebug1: control_persist_detach: backgrounding master process\\r\\ndebug2: control_persist_detach: background process is 22\\r\\ndebug2: fd 4 setting O_NONBLOCK\\r\\ndebug1: forking to background\\r\\ndebug1: Entering interactive session.\\r\\ndebug1: pledge: id\\r\\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\\r\\ndebug1: multiplexing control connection\\r\\ndebug2: fd 5 setting O_NONBLOCK\\r\\ndebug3: fd 5 is O_NONBLOCK\\r\\ndebug1: channel 1: new [mux-control]\\r\\ndebug3: channel_post_mux_listener: new mux channel 1 fd 5\\r\\ndebug3: mux_master_read_cb: channel 1: hello sent\\r\\ndebug2: set_control_persist_exit_time: cancel scheduled exit\\r\\ndebug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4\\r\\ndebug2: mux_master_process_hello: channel 1 client version 4\\r\\ndebug2: mux_client_hello_exchange: master version 4\\r\\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\\r\\ndebug3: mux_client_request_session: entering\\r\\ndebug3: mux_client_request_alive: entering\\r\\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4\\r\\ndebug2: mux_master_process_alive_check: channel 1: alive check\\r\\ndebug3: mux_client_request_alive: done pid = 24\\r\\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 75\\r\\ndebug2: mux_master_process_new_session: channel 1: request tty 0, X 0, agent 0, subsys 0, term "xterm", cmd "/bin/sh -c \\'echo ~root && sleep 0\\'", env 0\\r\\ndebug3: mux_client_request_session: session request sent\\r\\ndebug3: mux_master_process_new_session: got fds stdin 6, stdout 7, stderr 8\\r\\ndebug1: channel 2: new [client-session]\\r\\ndebug2: mux_master_process_new_session: channel_new: 2 linked to control channel 1\\r\\ndebug2: channel 2: send open\\r\\ndebug3: send packet: type 90\\r\\ndebug3: receive packet: type 80\\r\\ndebug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0\\r\\ndebug3: receive packet: type 4\\r\\ndebug1: Remote: /root/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\\r\\ndebug3: receive packet: type 4\\r\\ndebug1: Remote: /root/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\\r\\ndebug3: receive packet: type 91\\r\\ndebug2: channel_input_open_confirmation: channel 2: callback start\\r\\ndebug2: client_session2_setup: id 2\\r\\ndebug1: Sending command: /bin/sh -c \\'echo ~root && sleep 0\\'\\r\\ndebug2: channel 2: request exec confirm 1\\r\\ndebug3: send packet: type 98\\r\\ndebug3: mux_session_confirm: sending success reply\\r\\ndebug2: channel_input_open_confirmation: channel 2: callback done\\r\\ndebug2: channel 2: open confirm rwindow 0 rmax 32768\\r\\ndebug1: mux_client_request_session: master session id: 2\\r\\ndebug2: channel 2: rcvd adjust 2097152\\r\\ndebug3: receive packet: type 99\\r\\ndebug2: channel_input_status_confirm: type 99 id 2\\r\\ndebug2: exec request accepted on channel 2\\r\\ndebug3: receive packet: type 96\\r\\ndebug2: channel 2: rcvd eof\\r\\ndebug2: channel 2: output open -> drain\\r\\ndebug2: channel 2: obuf empty\\r\\ndebug2: chan_shutdown_write: channel 2: (i0 o1 sock -1 wfd 7 efd 8 [write])\\r\\ndebug2: channel 2: output drain -> closed\\r\\ndebug3: receive packet: type 98\\r\\ndebug1: client_input_channel_req: channel 2 rtype exit-status reply 0\\r\\ndebug3: mux_exit_message: channel 2: exit message, exitval 0\\r\\ndebug3: receive packet: type 98\\r\\ndebug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0\\r\\ndebug2: channel 2: rcvd eow\\r\\ndebug2: chan_shutdown_read: channel 2: (i0 o3 sock -1 wfd 6 efd 8 [write])\\r\\ndebug2: channel 2: input open -> closed\\r\\ndebug3: receive packet: type 97\\r\\ndebug2: channel 2: rcvd close\\r\\ndebug3: channel 2: will not send data after close\\r\\ndebug2: channel 2: send close\\r\\ndebug3: send packet: type 97\\r\\ndebug2: channel 2: is dead\\r\\ndebug2: channel 2: gc: notify user\\r\\ndebug3: mux_master_session_cleanup_cb: entering for channel 2\\r\\ndebug2: channel 1: rcvd close\\r\\ndebug2: channel 1: output open -> drain\\r\\ndebug2: chan_shutdown_read: channel 1: (i0 o1 sock 5 wfd 5 efd -1 [closed])\\r\\ndebug2: channel 1: input open -> closed\\r\\ndebug2: channel 2: gc: user detached\\r\\ndebug2: channel 2: is dead\\r\\ndebug2: channel 2: garbage collecting\\r\\ndebug1: channel 2: free: client-session, nchannels 3\\r\\ndebug3: channel 2: status: The following connections are open:\\r\\n  #1 mux-control (t16 nr0 i3/0 o1/16 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\\r\\n  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1)\\r\\n\\r\\ndebug2: channel 1: obuf empty\\r\\ndebug2: chan_shutdown_write: channel 1: (i3 o1 sock 5 wfd 5 efd -1 [closed])\\r\\ndebug2: channel 1: output drain -> closed\\r\\ndebug2: channel 1: is dead (local)\\r\\ndebug2: channel 1: gc: notify user\\r\\ndebug3: mux_master_control_cleanup_cb: entering for channel 1\\r\\ndebug2: channel 1: gc: user detached\\r\\ndebug2: channel 1: is dead (local)\\r\\ndebug3: mux_client_read_packet: read header failed: Broken pipe\\r\\ndebug2: channel 1: garbage collecting\\r\\ndebug1: channel 1: free: mux-control, nchannels 2\\r\\ndebug2: Received exit status from master 0\\r\\ndebug3: channel 1: status: The following connections are open:\\r\\n  #1 mux-control (t16 nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\\r\\n\\r\\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\\r\\n')
<2001:4178:6:1416:0000:000b:a:14> ESTABLISH SSH CONNECTION FOR USER: root
<2001:4178:6:1416:0000:000b:a:14> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/d04603209b"' 2001:4178:6:1416:0000:000b:a:14 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1718316863.213383-19-63963168782651 `" && echo ansible-tmp-1718316863.213383-19-63963168782651="` echo /root/.ansible/tmp/ansible-tmp-1718316863.213383-19-63963168782651 `" ) && sleep 0'"'"''

Actual results

Broken / Network unreachable

Identity added: /runner/artifacts/1144/ssh_key_data (/runner/artifacts/1144/ssh_key_data)
ansible-playbook [core 2.15.12]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /runner/artifacts/1144/callback/awx_display.py
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: CloudCheckSSH.yml ****************************************************
Positional arguments: CloudCheckSSH/CloudCheckSSH.yml
verbosity: 4
remote_user: shell
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/runner/inventory',)
subset: 2001:4178:6:1416:0000:000b:a:14
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in CloudCheckSSH/CloudCheckSSH.yml

PLAY [--> CloudCheckSSH] *******************************************************

TASK [Gathering Facts] *********************************************************
task path: /runner/project/CloudCheckSSH/CloudCheckSSH.yml:4
<2001:4178:6:1416:0000:000b:a:14> ESTABLISH SSH CONNECTION FOR USER: root
<2001:4178:6:1416:0000:000b:a:14> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/d04603209b"' 2001:4178:6:1416:0000:000b:a:14 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<2001:4178:6:1416:0000:000b:a:14> (255, b'', b'OpenSSH_8.7p1, OpenSSL 3.2.1 30 Jan 2024\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0000:000b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched \\'final\\'\\r\\ndebug2: match not found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug1: configuration requests final Match pass\\r\\ndebug2: resolve_canonicalize: hostname 2001:4178:6:1416:0000:000b:a:14 is address\\r\\ndebug2: resolve_canonicalize: canonicalised address "2001:4178:6:1416:0000:000b:a:14" => "2001:4178:6:1416:0:b:a:14"\\r\\ndebug1: re-parsing configuration\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for \\'final all\\' host 2001:4178:6:1416:0:b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched \\'final\\'\\r\\ndebug2: match found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts\\' -> \\'/root/.ssh/known_hosts\\'\\r\\ndebug3: expanded UserKnownHostsFile \\'~/.ssh/known_hosts2\\' -> \\'/root/.ssh/known_hosts2\\'\\r\\ndebug1: auto-mux: Trying existing master\\r\\ndebug1: Control socket "/runner/cp/d04603209b" does not exist\\r\\ndebug3: ssh_connect_direct: entering\\r\\ndebug1: Connecting to 2001:4178:6:1416:0:b:a:14 [2001:4178:6:1416:0:b:a:14] port 22.\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: connect to address 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable\\r\\nssh: connect to host 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable\\r\\n')
fatal: [2001:4178:6:1416:0000:000b:a:14]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: OpenSSH_8.7p1, OpenSSL 3.2.1 30 Jan 2024\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for 'final all' host 2001:4178:6:1416:0000:000b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'\\r\\ndebug2: match not found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug1: configuration requests final Match pass\\r\\ndebug2: resolve_canonicalize: hostname 2001:4178:6:1416:0000:000b:a:14 is address\\r\\ndebug2: resolve_canonicalize: canonicalised address \\"2001:4178:6:1416:0000:000b:a:14\\" => \\"2001:4178:6:1416:0:b:a:14\\"\\r\\ndebug1: re-parsing configuration\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config\\r\\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\\r\\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\\r\\ndebug2: checking match for 'final all' host 2001:4178:6:1416:0:b:a:14 originally 2001:4178:6:1416:0000:000b:a:14\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'\\r\\ndebug2: match found\\r\\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\\r\\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\\r\\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\\r\\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\\r\\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'\\r\\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'\\r\\ndebug1: auto-mux: Trying existing master\\r\\ndebug1: Control socket \\"/runner/cp/d04603209b\\" does not exist\\r\\ndebug3: ssh_connect_direct: entering\\r\\ndebug1: Connecting to 2001:4178:6:1416:0:b:a:14 [2001:4178:6:1416:0:b:a:14] port 22.\\r\\ndebug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x48\\r\\ndebug2: fd 3 setting O_NONBLOCK\\r\\ndebug1: connect to address 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable\\r\\nssh: connect to host 2001:4178:6:1416:0:b:a:14 port 22: Network is unreachable",
    "unreachable": true
}

PLAY RECAP *********************************************************************
2001:4178:6:1416:0000:000b:a:14 : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Additional information

I even did a tcpdump on the interface on my receptor node and i can see that when awx says Network is unreachable it DOES NOT try to connect. I can not see any tcp packet on the interface.

When it is working i see normal tcp packets on the interface.

I did some ICMP / manual SSH connection attempts and it always works. I run a ICMP test for about 60 minutes from the receptor node to the target server and no single package got dropped. Network is table and running without any issues.

discostur commented 4 months ago

Ok debugged it the last days and it seems to be related to slirp4netns. Maybe its a race condition or something like that. However, with podman 3 and slirp4netns 1.0.1 there seems to be many ipv6 issues (when you scroll through the github issues). Since my receptor node (and also podman) is running on debian 11 i won't get any newer versions with bugfixes.

However i found a workaround to disable slirp4netns and enable host networking. At the moment this cannot be configured on a job or template based option - you can just set it global via job settings (in awx ui): 

old: 

[
  "--network",
  "slirp4netns:enable_ipv6=true"
]

new: 
[
  "--network=host"
]

I found someone who was able to set it via extra_settings but that didn't work for me:

https://forum.ansible.com/t/awx-instance-receptor-podman-bug/2599/5 https://github.com/ansible/awx-operator/blob/devel/docs/user-guide/advanced-configuration/extra-settings.md

Podman then was always running a command like 

/usr/bin/podman run --rm --tty --interactive --workdir /runner/project -v /tmp/awx_1193_ycvbr77j/:/runner/:Z --env-file /tmp/awx_1193_ycvbr77j/artifacts/1193/env.list --quiet --name ansible_runner_1193 --user=root - - n e t w o r k = h o s t

which seems to be a syntax issue. However in the configmap on awx it look correct.

It would also be nice to configure such podman run options on a job or template base level and not just globally. It seems there are some tickets already open but nothing happend:

https://github.com/ansible/awx/issues/11552 https://github.com/ansible/awx/issues/12339

So for the moment it works for me via global job configuration (network=host).