Problem: Solution for Stuck ssh-mux processes

[NiceRath]

Versions

latest

Scope

Service (Job Scheduling, Job Preparation)

Issue

In some cases the ssh-mux processes spawned by Ansible get stuck. If that happens you cannot re-connect to the same host without killing that process manually. This is not an issue of Ansible-WebUI but of Ansible itself. However - it would be nice if that case would be handled by AW.

Example process:

root@ansible-webui:~# ps -aux | grep mux
ansible+  644620  0.3  0.1  14628  4016 ?        Ss   10:37   0:01 ssh: /home/ansible-webui/.ansible/cp/302932ae33 [mux]

Maybe add a button to manually kill all mux processes running longer than N minutes?

[NiceRath]

As a workaround we execute this script via an ansible-playbook:

aw_ps="$(pgrep -d, -u "$USER")"
to_kill="$(ps -p "$aw_ps" -o pid,etimes,cmd | grep '\[mux\]' | awk '{if($2>60) print $1}')"
kill $to_kill

It kills all mux processes running longer than 60sec. But this could kill good connections too. So I would only execute it manually if there's a need.

Playbook:

- name: Kill existing SSH-Mux processess
  hosts: localhost
  become: false
  gather_facts: false
  vars:
    min_run_time: 60

  tasks:
    - name: Pulling existing SSH-Mux Processes
      ansible.builtin.shell: |
        set -o pipefail
        aw_ps="$(pgrep -d, -u "$USER")"
        ps -p "$aw_ps" -o pid,etimes,cmd | grep '\[mux\]' | awk '{if($2>{{ min_run_time }}) print $1}' | tr '\n' ' '
      args:
        executable: '/bin/bash'
      register: ps_mux_existing
      changed_when: false

    - name: Will kill
      ansible.builtin.debug:
        var: ps_mux_existing.stdout
      when: ps_mux_existing.stdout != ""

    - name: Killing
      ansible.builtin.shell: |
        set -o pipefail
        echo "{{ ps_mux_existing.stdout }}" | xargs --no-run-if-empty kill
      args:
        executable: '/bin/bash'
      when: ps_mux_existing.stdout != ""