search

openshift / openshift-ansible

Install and config an OpenShift 3.x cluster
https://try.openshift.com
Apache License 2.0
2.19k stars 2.31k forks source link

Failures with ansible 2.2.1.0 #3111

Closed sdodson closed 7 years ago

sdodson commented 7 years ago

We're seeing yaml rendering issues with our code and 2.2.1.0. Need to sort out if we need to fix something or if ansible will be fixing things.

Upstream references https://github.com/ansible/ansible/issues/20290 https://github.com/ansible/ansible/issues/20253

    TASK [openshift_master : set_fact] *********************************************
    task path: /home/rdu/sdodson/git/Openshift/openshift-ansible/roles/openshift_master/tasks/main.yml:146
    An exception occurred during task execution. The full traceback is:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
        res = self._execute()
      File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 443, in _execute
        self._task.post_validate(templar=templar)
      File "/usr/lib/python2.7/site-packages/ansible/playbook/task.py", line 248, in post_validate
        super(Task, self).post_validate(templar)
      File "/usr/lib/python2.7/site-packages/ansible/playbook/base.py", line 373, in post_validate
        value = templar.template(getattr(self, name))
      File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 427, in template
        disable_lookups=disable_lookups,
      File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 383, in template
        disable_lookups=disable_lookups,
      File "/usr/lib/python2.7/site-packages/ansible/template/__init__.py", line 583, in do_template
        res = j2_concat(rf)
      File "<template>", line 9, in root
      File "/home/rdu/sdodson/git/Openshift/openshift-ansible/filter_plugins/openshift_master.py", line 488, in translate_idps
        return yaml.safe_dump([idp.to_dict() for idp in idp_list], default_flow_style=False)
      File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 218, in safe_dump
        return dump_all([data], stream, Dumper=SafeDumper, **kwds)
      File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 190, in dump_all
        dumper.represent(data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 28, in represent
        node = self.represent_data(data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 57, in represent_data
        node = self.yaml_representers[data_types[0]](self, data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 215, in represent_list
        return self.represent_sequence(u'tag:yaml.org,2002:seq', data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 101, in represent_sequence
        node_item = self.represent_data(item)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 57, in represent_data
        node = self.yaml_representers[data_types[0]](self, data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 223, in represent_dict
        return self.represent_mapping(u'tag:yaml.org,2002:map', data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping
        node_value = self.represent_data(item_value)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 67, in represent_data
        node = self.yaml_representers[None](self, data)
      File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 247, in represent_undefined
        raise RepresenterError("cannot represent an object: %s" % data)
    RepresenterError: cannot represent an object: htpasswd_auth

    fatal: [ose3-master.example.com]: FAILED! => {
        "failed": true,
        "msg": "Unexpected failure during module execution.",
        "stdout": ""
    }
sdodson commented 7 years ago

@abutcher had some proposed changes that may address this CC: @tdawson @detiber

abutcher commented 7 years ago

Casting ansible.vars.unsafe_proxy.AnsibleUnsafeText to string in the IdentityProviderBase class allowed this to continue. WIP commit here https://github.com/abutcher/openshift-ansible/commit/8d3714f7efe9833659a01ab37c27436ecac4b707

marekjelen commented 7 years ago

Hi, trying to install v1.4.0 cluster and Ansible is failing because of

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: RepresenterError: cannot represent an object: htpasswd_auth

I have tried to checkout release-1.4 branch as well as openshift-ansible-3.4.55-1 tag.

Is there some workaround that would allow me to finish the installation process except of downgrading Ansible itself?

thanks

detiber commented 7 years ago

@marekjelen I believe you would need to either need to use the commit that @abutcher linked to above. Ansible is working on a new build that resolves the problem, so I do not expect that we will ship a workaround.

abutcher commented 7 years ago

Playing whack-a-mole with openshift-ansible workarounds. The next place we hit the issue is within the to_padded_yaml filter.

TASK [openshift_master : Create master config] *********************************
task path: /home/abutcher/rhat/openshift-ansible/roles/openshift_master/tasks/main.yml:150
fatal: [master1.abutcher.com]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "backup": true, 
            "dest": "/etc/origin/master/master-config.yaml", 
            "group": "root", 
            "mode": 384, 
            "owner": "root", 
            "src": "master.yaml.v1.j2"
        }, 
        "module_name": "template"
    }, 
    "msg": "AnsibleFilterError: Failed to convert: cannot represent an object: BuildDefaults"
}
ArminMW commented 7 years ago

I can confirm that downgrading ansible-2.2.1.0-1.el7.noarch.rpm from epel-release-7-9 to ansible-2.2.0.0-4.el7.noarch.rpm from epel-release-7-8 is a workaround for a successful openshift-ansible v1.4 installation. Regards, Armin.

amdonov commented 7 years ago

@ArminMW Dumb question, but where did you find the old version? I'm not seeing it in EPEL.

wicksy commented 7 years ago

@amdonov Just run into exactly the same issue. Installed 2.2.0.0 as a pip.

ArminMW commented 7 years ago

I found the downlevel ansible in an epel mirror that seems to be lagging behind: http://ftp.uni-hannover.de/pub/mirror/linux/epel/7Server/x86_64/a/ansible-2.2.0.0-4.el7.noarch.rpm

amdonov commented 7 years ago

@wicksy Thanks for the tip. I took your advice and did something similar to the following. (Hope I'm not missing anything)

sudo yum -y erase ansible sudo yum install -y "@Development Tools" python2-pip openssl-devel python-devel sudo pip install -Iv ansible==2.2.0.0

Before going that route I tried applying the patch that closed the Ansible issue. I think that additional work will need to happen to the openshift playbooks to work going forward.

sdodson commented 7 years ago

We've also got ansible-2.2.0.0 in the centos paas sig common repo http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/common/ansible-2.2.0.0-1.el7.noarch.rpm If you choose to downgrade to 2.2.0.0 it should be noted that version is vulnerable to CVE-2016-9587, see https://bugzilla.redhat.com/show_bug.cgi?id=1404378

sdodson commented 7 years ago

We'll put a workaround in unless we're sure that we'll have a fixed version of ansible available soon.

leifmadsen commented 7 years ago

Just another confirmation that 2.2.0.0 downgrade results in a pass for me as well. Thanks for linking to the Ansible issues to track.

sdodson commented 7 years ago

Referenced PR is an incomplete fix fwiw.

ioggstream commented 7 years ago

Downgrading works. I strongly suggest:

anosulchik commented 7 years ago

+1 for ansible 2.2.1.1 in EPEL repo. thanks

detiber commented 7 years ago

@anosulchik are you saying you are seeing the same errors with 2.2.1.1?

anosulchik commented 7 years ago

@detiber Yes. Looks like EPEL repo now contains 2.2.1.0 as latest Ansible instead of 2.2.1.1. 2.2.1.0 works just fine.

jhcook commented 7 years ago

I've run in to this issue on macOS with virtualenv and sorted it by: pip install ansible==2.2.0.0 --upgrade

vbartoni commented 7 years ago

Would someone know if Ansible is willing to provide a fix for this in the near future (maybe with 2.2.2?), or something is possible to be done on this side? This issue is kind of painful - using openshift-on-openstack to install Origin, and fetching all packages from CentOS repos. Thanks!

sdodson commented 7 years ago

Yes, there should be an ansible release 2.2.2.0 or 2.2.1.1 soon, not sure which version they'll choose. Additionally, we're working to get a patched version into the Centos PaaS SIG repos. We'll be sure to update this when we know for sure when either of these lands.

On Tue, Feb 7, 2017 at 6:43 AM, Vedran Bartonicek notifications@github.com wrote:

Would someone know if Ansible is willing to provide a fix for this in the near future (maybe with 2.2.2?), or something is possible to be done on this side? This issue is kind of painful - using openshift-on-openstack to install Origin, and fetching all packages from CentOS repos. Thanks!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/openshift/openshift-ansible/issues/3111#issuecomment-277975242, or mute the thread https://github.com/notifications/unsubscribe-auth/AAC8IdGwjRc1v81WFx3DafUS5rCDxFnBks5raFjggaJpZM4LnLZI .

jospradlin commented 7 years ago

Just downgrade Ansible to from 2.2.1 to 2.2.0 using the YUM command below. (no other commands required)

# yum downgrade ansible

vbartoni commented 7 years ago

+1 thanks @jospradlin, great trick ;-)

vnugent commented 7 years ago

downgraded to ansible 2.2.0.0 but now I'm hitting https://github.com/ansible/ansible/issues/14143

env: Fedora 25 git describe: openshift-ansible-3.5.3-1-89-g405bd70

TASK [openshift_examples : Remove defunct quickstart templates from openshift namespace] ***
task path: /root/openshift-ansible/roles/openshift_examples/tasks/main.yml:90
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/commands/command.py
<172.18.17.19> ESTABLISH SSH CONNECTION FOR USER: root
<172.18.17.19> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAut
hentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -
o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r 172.18.17.19 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansib
le/tmp/ansible-tmp-1486578115.75-261880374043026 `" && echo ansible-tmp-1486578115.75-261880374043026="` echo $HOME/.ansible/t
mp/ansible-tmp-1486578115.75-261880374043026 `" ) && sleep 0'"'"''
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 91, in run
    item_results = self._run_loop(items)
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 274, in _run_loop
    res = self._execute(variables=task_vars)
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 490, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 574, in _execute_module
    tmp = self._make_tmp_path(remote_user)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 222, in _make_tmp_path
    result = self._low_level_execute_command(cmd, sudoable=False)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 754, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 670, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 598, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 434, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [172.18.17.19]: FAILED! => {
    "failed": true, 
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}
        to retry, use: --limit @/root/openshift-ansible/playbooks/byo/config.retry

PLAY RECAP *********************************************************************
172.18.17.19               : ok=265  changed=10   unreachable=0    failed=1   
172.18.17.27               : ok=60   changed=0    unreachable=0    failed=0   
localhost                  : ok=9    changed=0    unreachable=0    failed=0
DanyC97 commented 7 years ago

@sdodson can you please confirm if the ansible verion in centos-openshift-origin repo is been patch to cover the above issue?

in epel i see version 2.2.1.0-1.el7 in centos-openshift-origin repo i see 2.2.1.0-2.el7

sdodson commented 7 years ago

Yes, 2.2.1.0-2.el7 is patched, there may be openshift-ansible code paths that still have problems, if you're using 2.2.1.0-2.el7 and hit a problem with latest master lets get an issue open for those.

DanyC97 commented 7 years ago

sure will do indeed, thanks @sdodson !

OldCrowEW commented 7 years ago

pip install ansible==2.2.0.0 --upgrade fixes for me

esuomi commented 7 years ago

Hi,

We also were hit by this issue earlier this week with rather cryptic error:

TASK [openshift_master : set_fact] *********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: RepresenterError: cannot represent an object: claim
fatal: [10.27.41.51]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}

and indeed downgrading to 2.2.0.0 as @OldCrowEW suggested worked for us too.

jberkus commented 7 years ago

Incidentally, ansible 2.2.0.0 is no longer available as a package for Fedora 25. So at this point, there is no packaged version of ansible for Fedora 25 which works with Openshift-Ansible. (and replacing it via Pip is kinda painful, because it requires rebuilding the cryptography module).

DanyC97 commented 7 years ago

@jberkus if you read what i mentioned few comments above you can get the patched version from centos-openshift-origin and you are all good

jberkus commented 7 years ago

@DanyC97 well, I used a Fedora 24 container, which worked. But it does show that there's a deadline on fixing this one way or the other.

marconesns commented 7 years ago

Hello everyone,

It was like same problem, I decided to configure the master server in hours, in the case in a CentOS 7 with chrony. I have the version of ansbile 2.2.1.0

detiber commented 7 years ago

@jberkus @maxamillion the current situation is far from ideal by any means.

To give a bit more context:

I'm wondering if it would make sense to deliver OpenShift and openshift-ansible through a separate repo (or ideally repos, where each version is a separate repo) similar to how we are shipping in RHEL and CentOS. I think there are quite a few different reasons for why, but in this particular case, it would have allowed us to ship the patched Ansible version without affecting non-OpenShift users.

jberkus commented 7 years ago

Well, personally, I think we should ship an Openshift-ansible container which has all the bits.

detiber commented 7 years ago

@jberkus kind of like this: https://github.com/openshift/openshift-ansible/pull/3382 ?

There are some tricky problems with containerizing the install bits, though. The two biggest being:

The second reason is the biggest reason we haven't shipped something supported to date. Many users don't seem to like the idea of running the "installer" on a separate host and running a docker container on a host that you are managing the docker daemon and requiring a docker restart kind of kills your install.

jberkus commented 7 years ago

I generally solve the first problem by mounting my .ssh into the container. Not sure about proxy config, though, hasn't been an issue I had to solve.

Yes, self-bootstrapping by being able to run the installer on one of the target hosts would be ideal. However, until we solve that problem, a container we can run elsewhere would still be an improvement.

rastakajakwanna commented 7 years ago

I already use containerized Ansible-controller and it really needs a lot of tweaking to cover e.g. ansible-vault secrets, .ssh/{config,id_rsa} (ssh bastion host is not an issue), aws api credentials and all possible adjustments to /etc/ansible/ansible.cfg. However, it's worth it, especially in situations like this.

Btw, how do you solve such situations within Ansible Tower?? Would you be ever able to automate OCP installation with CloudForms and Tower?

PS: We just ran into this issue...

dougsaus commented 7 years ago

I'm having an issue and not sure if it is the same or not.

I am using: ansible.noarch 2.2.1.0-1.el7 @epel

Does this version have the fix or not?

The issue I am seeing is during: TASK [openshift_master : set_fact] Error message contains: cannot represent an object: allow_all

Is this the result of the same bug or do I have a unique issue?

sdodson commented 7 years ago

@Delmonte3161 That's the same problem. You can get ansible-2.2.1.0-2.el7 from the Centos PaaS SIG repo at http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/

jhadvig commented 7 years ago

@sdodson so I've installed the ansible-2.2.1.0-2.el7 from repo you suggested but I still can see the issue

TASK [openshift_master : set_fact] ***** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: RepresenterError: cannot represent an object: allow_all fatal: [localhost]: FAILED! => { "failed": true } MSG: Unexpected failure during module execution.

jberkus commented 7 years ago

@rastakajakwanna oh, hey, how did you deal with secrets?

rastakajakwanna commented 7 years ago

@jberkus some of them are parsed from known locations and exposed as env vars in the container, ansible-vault secret is mounted to the container user home location, that's the easiest solution.

jdeglopper commented 7 years ago

For Fedora 25, ansible 2.2.2.0 is available in Fedora Rawhide as ansible-2.2.2.0-0.1.rc1.fc26.noarch.rpm , which appears to correctly run the playbook.

sdodson commented 7 years ago

Yes, as soon as 2.2.2.0 is available we can put this whole mess behind us. Unfortunately that's happened slower than expected.

mkungla commented 7 years ago

For Fedora 25 had to go for rawhide too which solved the issue. 

dnf install -y fedora-repos-rawhide
dnf install -y --enablerepo rawhide --best --allowerasing ansible
sdodson commented 7 years ago

Ansible 2.2.2.0 has been released, it should be available in F25 and EPEL shortly. OCP channels will be updated after we've validated this version.

jberkus commented 7 years ago

Looking forward to it. When it's available, we can close this.

x1957 commented 7 years ago

same problem with ansible 2.2.2.0

An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud02.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<c4-container-cloud06.bj> ESTABLISH SSH CONNECTION FOR USER: root
<c4-container-cloud06.bj> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r c4-container-cloud06.bj '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<c4-container-cloud07.bj> ESTABLISH SSH CONNECTION FOR USER: root
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud04.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
<c4-container-cloud07.bj> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r c4-container-cloud07.bj '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud03.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<c4-container-cloud08.bj> ESTABLISH SSH CONNECTION FOR USER: root
<c4-container-cloud08.bj> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r c4-container-cloud08.bj '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud05.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud07.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud06.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud08.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 126, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 502, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 33, in run
    results = merge_hash(results, self._execute_module(tmp=tmp, task_vars=task_vars))
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 653, in _execute_module
    res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 777, in _low_level_execute_command
    rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 674, in exec_command
    return_tuple = self._exec_command(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 599, in _exec_command
    (returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py", line 437, in _run
    rfd, wfd, efd = select.select(rpipes, [], [], timeout)
ValueError: filedescriptor out of range in select()
fatal: [c4-container-cloud01.bj]: FAILED! => {
    "failed": true
}

MSG:

Unexpected failure during module execution.
sdodson commented 7 years ago

@x1957 Can you please open a new issue for that, that doesn't look like any of the other errors in this issue and it'd be helpful to gather the output with -vvvv option and include the tasks leading up to this failure.