SSH connection works for `ansible -m` but not for `ansible-playbook`, version 2.2.0

[fvant]

ISSUE TYPE

• Bug Report

  COMPONENT NAME

SSH connection

ANSIBLE VERSION

ansible 2.2.0 (devel eb33ed4219) last updated 2016/09/27 11:16:30 (GMT +200)
  lib/ansible/modules/core: (detached HEAD c03697c81e) last updated 2016/09/27 11:17:10 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD 119bc466be) last updated 2016/09/27 11:17:58 (GMT +200)
  config file = 
  configured module search path = Default w/o overrides

CONFIGURATION

N/A

OS / ENVIRONMENT

Linux localhost 3.8.11 #1 SMP Mon Sep 12 21:56:54 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Python 2.7.11
OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6, OpenSSL 1.0.1f 6 Jan 2014

~/ansible$ pip list ansible (2.2.0) apache-libcloud (0.20.1) aws (0.2.5) awscli (1.10.61) backports.ssl-match-hostname (3.5.0.1) boto (2.40.0) boto3 (1.1.3) botocore (1.4.51) certifi (2015.11.20.1) cffi (1.7.0) colorama (0.3.7) Cython (0.20.1.post0) docutils (0.12) ecdsa (0.13) enum34 (1.1.6) Fabric (1.11.1) futures (3.0.5) google-api-python-client (1.1) httplib2 (0.8) idna (2.1) ipaddress (1.0.16) Jinja2 (2.8) jmespath (0.9.0) jsonpatch (1.13) jsonpointer (1.10) jwt (0.3.2) lxml (3.3.3) MarkupSafe (0.23) numpy (1.8.2) paramiko (1.17.0) pip (8.1.2) ply (3.4) prettytable (0.7.2) pyasn1 (0.1.9) pycparser (2.14) pycrypto (2.6.1) pygpgme (0.3) pyOpenSSL (0.13) python-apt (0.9.3.5ubuntu2) python-dateutil (2.5.3) python-gflags (3.0.4) python-nghttp2 (1.3.5.dev0) PyYAML (3.11) rsa (3.4.2) s3transfer (0.1.3) setuptools (23.0.0) simplejson (3.3.1) six (1.10.0) tabulate (0.7.5) urllib3 (1.14) virtualenv (1.11.4)

SUMMARY

Using a repo installed version of Ansible, 2.1.1 all works fine.

When I remove that version and then clone the devel branch, ansible-playbook fails to connect to hosts while running ansible -m setup consistently works and gets facts from the host.

STEPS TO REPRODUCE

ansible mng -m setup -a 'gather_subset=!all' -vvvvvv

Using /home/fvant/ansible-local/ansible.cfg as config file
Loading callback plugin minimal of type stdout, v2.0 from /home/fvant/ansible/lib/ansible/plugins/callback/__init__.pyc
Using module file /home/fvant/ansible/lib/ansible/modules/core/system/setup.py
<10.0.91.104> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.91.104> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<10.0.91.104> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<10.0.91.104> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="./vars/keys/ansible_id_rsa")
<10.0.91.104> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<10.0.91.104> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<10.0.91.104> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=5)
<10.0.91.104> SSH: PlayContext set ssh_common_args: ()
<10.0.91.104> SSH: PlayContext set ssh_extra_args: ()
<10.0.91.104> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r)
<10.0.91.104> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="./vars/keys/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r 10.0.91.104 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1474979078.18-42020272005162 `" && echo ansible-tmp-1474979078.18-42020272005162="` echo /tmp/ansible-tmp-1474979078.18-42020272005162 `" ) && sleep 0'"'"''
<10.0.91.104> PUT /tmp/tmpUCSQuH TO /tmp/ansible-tmp-1474979078.18-42020272005162/setup.py
<10.0.91.104> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<10.0.91.104> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<10.0.91.104> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="./vars/keys/ansible_id_rsa")
<10.0.91.104> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<10.0.91.104> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<10.0.91.104> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=5)
<10.0.91.104> SSH: PlayContext set ssh_common_args: ()
<10.0.91.104> SSH: PlayContext set sftp_extra_args: ()
<10.0.91.104> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r)
<10.0.91.104> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="./vars/keys/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r '[10.0.91.104]'
<10.0.91.104> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.91.104> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<10.0.91.104> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<10.0.91.104> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="./vars/keys/ansible_id_rsa")
<10.0.91.104> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<10.0.91.104> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<10.0.91.104> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=5)
<10.0.91.104> SSH: PlayContext set ssh_common_args: ()
<10.0.91.104> SSH: PlayContext set ssh_extra_args: ()
<10.0.91.104> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r)
<10.0.91.104> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="./vars/keys/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r 10.0.91.104 '/bin/sh -c '"'"'chmod u+x /tmp/ansible-tmp-1474979078.18-42020272005162/ /tmp/ansible-tmp-1474979078.18-42020272005162/setup.py && sleep 0'"'"''
<10.0.91.104> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.91.104> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<10.0.91.104> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<10.0.91.104> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="./vars/keys/ansible_id_rsa")
<10.0.91.104> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<10.0.91.104> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<10.0.91.104> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=5)
<10.0.91.104> SSH: PlayContext set ssh_common_args: ()
<10.0.91.104> SSH: PlayContext set ssh_extra_args: ()
<10.0.91.104> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r)
<10.0.91.104> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="./vars/keys/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r -tt 10.0.91.104 '/bin/sh -c '"'"'/usr/bin/python /tmp/ansible-tmp-1474979078.18-42020272005162/setup.py; rm -rf "/tmp/ansible-tmp-1474979078.18-42020272005162/" > /dev/null 2>&1 && sleep 0'"'"''
mng-a-sandbox01 | SUCCESS => {
    "ansible_facts": {
        "ansible_architecture": "x86_64", 
        "ansible_cmdline": {
            "BOOT_IMAGE": "/boot/vmlinuz-3.10.0-327.10.1.el7.x86_64", 
            "LANG": "en_US.UTF-8", 
            "console": "ttyS0,115200", 
            "crashkernel": "auto", 
            "ro": true, 
            "root": "UUID=ef6ba050-6cdc-416a-9380-c14304d0d206"
        }, 

</snip>

EXPECTED RESULTS

ACTUAL RESULTS

ansible-playbook  playbooks/main.yml -e env=mng --ask-vault-pass -e account=playground -vvvvvvvvv

Loading callback plugin default of type stdout, v2.0 from /home/fvant/ansible/lib/ansible/plugins/callback/__init__.pyc
Loading callback plugin profile_tasks of type aggregate, v2.0 from /home/fvant/ansible/lib/ansible/plugins/callback/__init__.pyc

PLAYBOOK: main.yml *************************************************************
11 plays in playbooks/main.yml

PLAY [mng] *********************************************************************

TASK [setup] *******************************************************************
Tuesday 27 September 2016  14:19:31 +0200 (0:00:00.177)       0:00:01.441 ***** 
Using module file /home/fvant/ansible/lib/ansible/modules/core/system/setup.py
<10.0.91.104> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.91.104> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<10.0.91.104> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<10.0.91.104> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="./vars/keys/ansible_id_rsa")
<10.0.91.104> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<10.0.91.104> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=root)
<10.0.91.104> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=5)
<10.0.91.104> SSH: PlayContext set ssh_common_args: ()
<10.0.91.104> SSH: PlayContext set ssh_extra_args: ()
<10.0.91.104> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r)
<10.0.91.104> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="./vars/keys/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r 10.0.91.104 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1474978771.34-100536752987114 `" && echo ansible-tmp-1474978771.34-100536752987114="` echo /tmp/ansible-tmp-1474978771.34-100536752987114 `" ) && sleep 0'"'"''
fatal: [mng-a-sandbox01]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh.", 
    "unreachable": true
}

[Shaps]

@fvant Have you tried running the ssh command from the shell to see if you can actually connect to the server? ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="./vars/keys/ansible_id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=5 -o ControlPath=/home/fvant/.ansible/cp/ansible-ssh-%h-%p-%r 10.0.91.104 'date'

I see you are connecting as root in the failed play, do you have ssh root access to that server?

[fvant]

sure, the ansible -m setup also runs as root

debug2: channel 2: rcvd eow
debug2: channel 2: close_read
debug2: channel 2: input open -> closed
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
Tue Sep 27 15:19:29 CEST 2016
debug3: channel 2: will not send data after close
debug2: channel 2: obuf empty
debug2: channel 2: close_write

[dav1x]

+1 Having this exact same issue. Ansible-playbook fails to connect via ssh. However, if I connect ahead of time via an ssh the control master is set and everything works fine.

I resolved by removing ansible 2.2 and installed 2.1.

[Shaps]

I don't seem to be able to reproduce this, can you please provide the playbook where this error happens?

[agaffney]

The first ssh command in the ad-hoc command and the first failing ssh command in the playbook run appear to be identical, aside from the name of the temp dir.

[agaffney]

It would be very helpful if you could provide a small testcase. The playbook that you are using has 11 plays in it, which doesn't really make it good for other people trying to reproduce this issue.

[fvant]

I execute the commands directly after each other from the same directory.

completely removing 2.2 and a yum install ansible, results in both commands working.

On Thu, Sep 29, 2016 at 6:28 PM, Andrew Gaffney notifications@github.com wrote:

The first ssh command in the ad-hoc command and the first failing ssh command in the playbook run appear to be identical, aside from the name of the temp dir.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ansible/ansible/issues/17770#issuecomment-250518931, or mute the thread https://github.com/notifications/unsubscribe-auth/AFE7QxqfP1ioV7FebG4H9kDDjWo4QsYVks5qu-cugaJpZM4KHm_z .

[agaffney]

You execute what commands directly? What are the results when you execute those commands?

Edit: I assume you're referring to the ansible and ansible-playbook commands

[fvant]

I will try to set up an example playbook but the playbook fails at the gather_facts Setup task, so any playbook will do I guess

@agaffney yes I meant those

On Thu, Sep 29, 2016 at 6:31 PM, Andrew Gaffney notifications@github.com wrote:

It would be very helpful if you could provide a small testcase. The playbook that you are using has 11 plays in it, which doesn't really make it good for other people trying to reproduce this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ansible/ansible/issues/17770#issuecomment-250519812, or mute the thread https://github.com/notifications/unsubscribe-auth/AFE7QwOan3IjBeHVPQ0CEaNKalm8dWyHks5qu-fogaJpZM4KHm_z .

[agaffney]

It's possibly that a vars: section in the playbook or group_vars/ in the same dir as the playbook are having an effect here. Make sure that your example playbook takes these things into account.

[winglian]

I'm also seeing the same issue when using ansible-playbook with v2.1.2.0 that "resolves" itself when going back to v2.1.1.0.

[Shaps]

Can you also please specify what version of OS are you using? I'm still not able to replicate this, I've tested on Ubuntu 14.04 and Ubuntu 16.06

[fvant]

Several, Centos 7 and Ubuntu Trusty (14.04.5 LTS)

On Fri, Sep 30, 2016 at 9:15 AM, Andrea Tartaglia notifications@github.com wrote:

Can you also please specify what version of OS are you using? I'm still not able to replicate this, I've tested on Ubuntu 14.04 and Ubuntu 16.06

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ansible/ansible/issues/17770#issuecomment-250676204, or mute the thread https://github.com/notifications/unsubscribe-auth/AFE7Q0OceWCxUqZUGleWjfssK71QYs2Rks5qvLckgaJpZM4KHm_z .

[Shaps]

@fvant Ok, can you still provide a test case playbook ( including any eventual vars your playbook might be including )

[fvant]

I am trying to get to the minimal failing play, so far I have ruled out the use of roles and ansible-vault as making a breaking difference in my case. But the failing playbook has a lot of includes and gives warnings like this:

 [WARNING]: While constructing a mapping from /home/fvant/ansible-local/playbooks/main.yml, line 8, column 3, found a duplicate
dict key (vars_files). Using last defined value only.

statically included: /home/fvant/ansible-local/roles/newrelic_server/tasks/install.yml
..more..
statically included: /home/fvant/ansible-local/playbooks/mng-vpn.yml
 [WARNING]: While constructing a mapping from /home/fvant/ansible-local/playbooks/mng-rundeck.yml, line 38, column 3, found a
duplicate dict key (file). Using last defined value only.

Interesting is perhaps that a working run with 2.1.1 gives the same warnings but not the "statically included" lines

Some of them are meant to be dynamic and does 2.2 introduce a different default behaviour here ?

[omgjlk]

I was able to replicate this. It is a weird one.

First, I have a custom ssh_config file that I use, and I was using ANSIBLE_SSH_ARGS="-F tmp/ssh_config" to point to it.

Next, when I was testing, my playbook was outside of the current working directory, playbooks/tests/tasks/foo.yaml.

This combo does not work, as of commit to fix bug #16805 ( 9255a618e307b4e087cb0f3f6f4f225f7652c298 ), as that resets the cwd to be relative to the playbook file.

If I move my playbook to a place where the path to my ssh config works out, then everything is fine.

Obviously I'll be changing my setup here so that the path to the ssh config is no longer relatively referenced, but I wonder what else will break because of this behavior change.

[cadavre]

As described – it also affects 2.1.2.0.