"ERROR! A worker was found in a dead state" with libssh connection plugin

SUMMARY

When using ansible_connection: ansible.netcommon.libssh in playbook, the playbook will crash with "ERROR! A worker was found in a dead state" message.

ISSUE TYPE

Bug Report

COMPONENT NAME

ansible.netcommon.libssh

ANSIBLE VERSION

[root@ef1a4a8f452a app]# ansible --version
ansible 2.10.6
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]

CONFIGURATION

[root@ef1a4a8f452a app]# ansible-config dump --only-changed

(None changed)

OS / ENVIRONMENT

Official Docker centos:7 image.

STEPS TO REPRODUCE

Start a CentOS 7 docker: docker run -it -v $(pwd):/usr/src/app centos:7 bash

Install minimal amount of packages:

yum install -y sshpass vim libssh-devel python3
python3 -m pip install -U pip wheel
python3 -m pip install ansible ansible-pylibssh
ansible-galaxy collection install ansible.netcommon

Execute playbook: ansible-playbook -i inventory.yml playbook.yml
Get "ERROR! A worker was found in a dead state" message

Playbook:

- hosts: all
  gather_facts: false
  vars:
    ansible_connection: ansible.netcommon.libssh
  tasks:
  - name: Echo 123
    shell: "echo 123"

Inventory:

ipa_integrated:
  hosts:
    192.168.11.203:
  vars:
    ansible_user: eric.wong
    ansible_ssh_pass: some_password

EXPECTED RESULTS

The playbook should complete normally with normal end-of-playbook summary.

ACTUAL RESULTS

The message "ERROR! A worker was found in a dead state" is displayed and the playbook execution is interrupted.

[root@ef1a4a8f452a app]# ansible-playbook -i inventory.yml playbook.yml -vvvvv
ansible-playbook 2.10.6
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
No config file found; using defaults
setting up inventory plugins
host_list declined parsing /usr/src/app/inventory.yml as it did not pass its verify_file() method
Parsed /usr/src/app/inventory.yml inventory source with yaml plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.6/site-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: playbook.yml **********************************************************************************************************************************************************************************************************************************
Positional arguments: playbook.yml
verbosity: 5
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/usr/src/app/inventory.yml',)
forks: 5
1 plays in playbook.yml

PLAY [all] **********************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [Echo 123] *****************************************************************************************************************************************************************************************************************************************
task path: /usr/src/app/playbook.yml:6
Loading collection ansible.netcommon from /root/.ansible/collections/ansible_collections/ansible/netcommon
<192.168.11.203> ESTABLISH LIBSSH CONNECTION FOR USER: eric.wong on PORT 22 TO 192.168.11.203
[2021/03/15 05:09:52.297431, 2] ssh_connect:  libssh 0.9.3 (c) 2003-2019 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
[2021/03/15 05:09:52.299148, 2] ssh_socket_connect:  Nonblocking connection socket: 9
[2021/03/15 05:09:52.300367, 2] ssh_connect:  Socket connecting, now waiting for the callbacks to work
[root@ef1a4a8f452a app]# 
[root@ef1a4a8f452a app]# 
[root@ef1a4a8f452a app]# 
[root@ef1a4a8f452a app]# ansible-playbook -i inventory.yml playbook.yml -vvvvv
ansible-playbook 2.10.6
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
No config file found; using defaults
setting up inventory plugins
host_list declined parsing /usr/src/app/inventory.yml as it did not pass its verify_file() method
Parsed /usr/src/app/inventory.yml inventory source with yaml plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.6/site-packages/ansible/plugins/callback/default.py
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: playbook.yml **********************************************************************************************************************************************************************************************************************************
Positional arguments: playbook.yml
verbosity: 5
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/usr/src/app/inventory.yml',)
forks: 5
1 plays in playbook.yml

PLAY [all] **********************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [Echo 123] *****************************************************************************************************************************************************************************************************************************************
task path: /usr/src/app/playbook.yml:6
Loading collection ansible.netcommon from /root/.ansible/collections/ansible_collections/ansible/netcommon
<192.168.11.203> ESTABLISH LIBSSH CONNECTION FOR USER: eric.wong on PORT 22 TO 192.168.11.203
[2021/03/15 05:13:31.652056, 2] ssh_connect:  libssh 0.9.3 (c) 2003-2019 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
[2021/03/15 05:13:31.652674, 2] ssh_socket_connect:  Nonblocking connection socket: 9
[2021/03/15 05:13:31.653033, 2] ssh_connect:  Socket connecting, now waiting for the callbacks to work
[2021/03/15 05:13:31.732595, 1] socket_callback_connected:  Socket connection callback: 1 (0)
[2021/03/15 05:13:31.832018, 2] ssh_client_connection_callback:  SSH server banner: SSH-2.0-OpenSSH_7.4
[2021/03/15 05:13:31.832150, 2] ssh_analyze_banner:  Analyzing banner: SSH-2.0-OpenSSH_7.4
[2021/03/15 05:13:31.832188, 2] ssh_analyze_banner:  We are talking to an OpenSSH client version: 7.4 (70400)
[2021/03/15 05:13:31.832760, 1] ssh_known_hosts_read_entries:  Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
[2021/03/15 05:13:31.912600, 2] ssh_kex_select_methods:  Negotiated curve25519-sha256,ssh-ed25519,aes256-gcm@openssh.com,aes256-gcm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-256-etm@openssh.com,none,none,,
[2021/03/15 05:13:32.012623, 2] ssh_init_rekey_state:  Set rekey after 4294967296 blocks
[2021/03/15 05:13:32.012751, 2] ssh_init_rekey_state:  Set rekey after 4294967296 blocks
[2021/03/15 05:13:32.012805, 2] ssh_packet_client_curve25519_reply:  SSH_MSG_NEWKEYS sent
[2021/03/15 05:13:32.012880, 2] ssh_packet_newkeys:  Received SSH_MSG_NEWKEYS
[2021/03/15 05:13:32.013176, 2] ssh_packet_newkeys:  Signature verified and valid
[2021/03/15 05:13:32.204546, 1] ssh_packet_userauth_failure:  Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
[2021/03/15 05:13:32.204634, 2] ssh_packet_userauth_failure:  Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
ssh connection is OK: <pylibsshext.session.Session object at 0x7f4952a980f8>
[2021/03/15 05:13:33.722675, 2] channel_open:  Creating a channel 43 with 64000 window and 32768 max packet
[2021/03/15 05:13:33.824130, 1] ssh_packet_process:  Couldn't do anything with packet type 80
[2021/03/15 05:13:33.919541, 2] ssh_packet_channel_open_conf:  Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
[2021/03/15 05:13:33.919635, 2] ssh_packet_channel_open_conf:  Remote window : 0, maxpacket : 32768
<192.168.11.203> EXEC /bin/sh -c 'echo ~eric.wong && sleep 0'
[2021/03/15 05:13:34.021007, 2] channel_rcv_change_window:  Adding 2097152 bytes to channel (43:0) (from 0 bytes)
[2021/03/15 05:13:34.021150, 2] channel_request:  Channel request exec success
[2021/03/15 05:13:34.062766, 2] grow_window:  growing window (channel 43:0) to 1280000 bytes
[2021/03/15 05:13:34.066835, 2] channel_open:  Creating a channel 44 with 64000 window and 32768 max packet
ERROR! A worker was found in a dead state

ansible-collections / ansible.netcommon

"ERROR! A worker was found in a dead state" with libssh connection plugin #237