Open diioc opened 5 years ago
This looks like it should be a simple fix -- however I am worried about AIX target :) None of this code is tested on AIX
I notice some issues:
If you log into the system, does cutpasting that command line produce a password prompt? su db2inst1 -c "/usr/bin/python -c \"print 123\""
If it does not, is it possible to check the system log? AIX is rejecting the authentication for some reason -- it will not even ask for a password. This suggests something about Mitogen's process environment is incorrect, and as a result, something like PAM is refusing to do business
It is possible your missing 'become_flags' is causing the problem, but at least for now I doubt it.
From googling "authentication is denied", it may relate to AIX RBAC authentication
I tried to execute su db2inst1 -c "/usr/bin/python -c \"print 123\""
in multiple different ways:
sudo su db2inst1 -c "/usr/bin/python -c \"print 123\""
=> prompts for a sudo passI'll try to reproduce it on rhel7
If it works when you are logged in interactively, then it must be some problem with how Mitogen creates processes or TTYs, or perhaps because Mitogen launches a non-interactive SSH session, it does not receive some important privilege.
I'm afraid I do not know enough about AIX to understand this issue :) Do you know of an expert within your organization that would understand deeper security issues? I am fairly sure this is some strange security restriction AIX has, and to make things work, we must meet those restriction.
Hi man! Guess what?) I reproduced the issue on rhel7. It has a different output in same situation but I can fix it by the same way to replace 'become: yes' statement. It don't work It works fine
Logs from node mitogen.13780.20190212_172029.log mitogen.13771.20190212_172028.log
And output from host:
ansible-playbook mit_test.yml -i inventory -vvv --limit node01
ansible-playbook 2.6.1
config file = /Users/diioc/devops/vagrant/rhel-db2/provisioning/ansible.cfg
configured module search path = [u'/Users/diioc/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
executable location = /usr/local/bin/ansible-playbook
python version = 2.7.15 (default, Jun 17 2018, 12:46:58) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
Using /Users/diioc/devops/vagrant/rhel-db2/provisioning/ansible.cfg as config file
Parsed /Users/diioc/devops/vagrant/rhel-db2/provisioning/inventory inventory source with ini plugin
PLAYBOOK: mit_test.yml **********************************************************************************
1 plays in mit_test.yml
PLAY [node] *********************************************************************************************
[pid 53505] 17:20:25.865219 D mitogen: mitogen.service.Pool(0x1119483d0, size=16, th='MainThread'): initialized
[pid 53505] 17:20:25.870836 D ansible_mitogen.process: Service pool configured: size=16
META: ran handlers
TASK [command] ******************************************************************************************
task path: /Users/diioc/devops/vagrant/rhel-db2/provisioning/mit_test.yml:10
[pid 53506] 17:20:25.916618 D mitogen: unix.connect(path='/var/folders/pf/rc9rjb7x2_1d3r_73cq5l0w80000gn/T/mitogen_unix_wPEhwd')
[pid 53506] 17:20:25.918792 D mitogen: unix.connect(): local ID is 1, remote is 0
[pid 53505] 17:20:25.918860 D mitogen: register(Context(1, None), mitogen.core.Stream(u'unix_client.53506'))
[pid 53505] 17:20:25.928756 D mitogen: mitogen.ssh.Stream(u'default').connect()
[pid 53505] 17:20:26.096152 D mitogen: create_child() child 53507 fd 95, parent 53505, cmd: ssh -o "LogLevel ERROR" -l vagrant -o "Compression yes" -o "ServerAliveInterval 15" -o "ServerAliveCountMax 3" -o "BatchMode yes" -o "StrictHostKeyChecking yes" -C -o ControlMaster=auto -o ControlPersist=60s 192.168.33.12 /usr/bin/python -c "'import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkVFLwzAUhZ/XX9G3JCxuycYQCgFlD+KDCEXsgw5Jm1SDWRLSbnX+eu86Ye188O1+3HNzDic5LYRvZsEEjUkSaTcgU6cAtY+fmGTJBGa1CwvMKGeMnDmnQ4qw5SeurG80zocQh1AMoQMAw+YA9la24LpNhUiRkrEzDqXSqX6pv3S1a2Vpdb+e75o4L42bh0P74R2CnJML2VT0h3sdG+PdS7bc9Lba7U0ERrf53TNDGzE+O2kALR4v6BinCG9N69+1y5QxvrqRjbcHZ9yV2cpqZn0lbbZartiKIJLAm100rcacoof7p0fG2KtDkKXyCionyVq84WPpygftoGoUS0RmUUuF+eKac0LRtwnwUh3EWVdQ1JXo+A91+DVY9/Op2wt195/6b0o+SvkDRZeyBA==\".encode(),\"base64\"),\"zip\"))'"
[pid 53505] 17:20:26.098825 D mitogen: mitogen.ssh.Stream(u'local.53507').connect(): child process stdin/stdout=95
[pid 53505] 17:20:26.508275 D mitogen: mitogen.ssh.Stream(u'local.53507'): received 'MITO000\n'
[pid 53505] 17:20:26.509444 D mitogen: mitogen.ssh.Stream(u'local.53507')._ec0_received()
[pid 53505] 17:20:26.525306 D mitogen: register(Context(2, u'ssh.192.168.33.12'), mitogen.ssh.Stream(u'ssh.192.168.33.12'))
[pid 53505] 17:20:26.530016 D mitogen: CallChain(Context(2, u'ssh.192.168.33.12')).call_async(): ansible_mitogen.target.init_child(candidate_temp_dirs=[u'/tmp/.ansible-${USER}/tmp', u'/var/tmp', u'/tmp'], log_level=10)
[pid 53505] 17:20:26.531752 D mitogen: Context(2, u'ssh.192.168.33.12').send_async(Message(2, 0, 0, 101, 1000, '\x80\x02(NX\x16\x00\x00\x00ansible_mitogen.targetNX\n\x00\x00\x00init_child)cm'..185))
[pid 53505] 17:20:26.540609 D mitogen: _build_tuple('/usr/local/lib/python2.7/site-packages/ansible/__init__.py', u'ansible') -> [u'cli', u'compat', u'config', u'constants', u'errors', u'executor', u'galaxy', u'inventory', u'module_utils', u'modules', u'parsing', u'playbook', u'plugins', u'release', u'template', u'utils', u'vars']
[pid 53505] 17:20:26.542824 D mitogen: _send_load_module(mitogen.ssh.Stream(u'ssh.192.168.33.12'), u'ansible.release')
[pid 53505] 17:20:26.544781 D mitogen: _send_load_module(mitogen.ssh.Stream(u'ssh.192.168.33.12'), u'ansible')
[pid 53505] 17:20:26.550907 D mitogen: _build_tuple('/usr/local/lib/python2.7/site-packages/ansible/module_utils/__init__.py', u'ansible.module_utils') -> [u'_text', u'acme', u'ansible_release', u'ansible_tower', u'api', u'aws', u'azure_rm_common', u'azure_rm_common_rest', u'basic', u'cloud', u'cloudscale', u'cloudstack', u'common', u'connection', u'crypto', u'database', u'digital_ocean', u'dimensiondata', u'docker_common', u'ec2', u'exoscale', u'f5_utils', u'facts', u'firewalld', u'gcdns', u'gce', u'gcp', u'gcp_utils', u'gitlab', u'heroku', u'infinibox', u'influxdb', u'ipa', u'ismount', u'json_utils', u'k8s', u'keycloak', u'known_hosts', u'ldap', u'lxd', u'manageiq', u'memset', u'mysql', u'net_tools', u'netapp', u'network', u'oneandone', u'oneview', u'opennebula', u'openstack', u'ovirt', u'parsing', u'postgres', u'powershell', u'pure', u'pycompat24', u'rax', u'redhat', u'remote_management', u'scaleway', u'service', u'six', u'splitter', u'univention_umc', u'urls', u'vca', u'vmware', u'vmware_rest_client', u'vultr']
...
...
Importer().find_module('ansible.module_utils.parsing')
[pid 53505] 17:20:27.499786 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer.load_module(u'ansible.module_utils.parsing')
[pid 53505] 17:20:27.500749 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('ansible.module_utils.parsing.convert_bool')
[pid 53505] 17:20:27.501596 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer.load_module(u'ansible.module_utils.parsing.convert_bool')
[pid 53505] 17:20:27.502350 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('ansible.module_utils.parsing.ansible')
[pid 53505] 17:20:27.503102 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer(): master doesn't know 'ansible.module_utils.parsing.ansible'
[pid 53505] 17:20:27.503874 D mitogen.ctx.ssh.192.168.33.12: ansible_mitogen.runner: EnvironmentFileWatcher(u'/home/vagrant/.pam_environment') installed; existing keys: []
[pid 53505] 17:20:27.504614 D mitogen.ctx.ssh.192.168.33.12: ansible_mitogen.runner: EnvironmentFileWatcher(u'/etc/environment') installed; existing keys: []
[pid 53505] 17:20:27.505436 D mitogen.ctx.ssh.192.168.33.12: mitogen: replaced Poller(0x7f32066a35d0) with EpollPoller(0x7f31faddf910) (new: 4 readers, 1 writers; old: 4 readers, 1 writers)
[pid 53505] 17:20:27.506212 D mitogen.ctx.ssh.192.168.33.12: mitogen: Router(Broker(0x7f32066a3490)).upgrade()
[pid 53505] 17:20:27.507037 D mitogen: IdAllocator(Router(Broker(0x11191ea90))): allocating [3..1003)
[pid 53505] 17:20:27.507593 D mitogen: IdAllocator(Router(Broker(0x11191ea90))): allocating [3..1003) to Context(2, u'ssh.192.168.33.12')
[pid 53505] 17:20:27.529339 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.fork.Stream(u'default').connect()
[pid 53505] 17:20:27.530358 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.fork.Stream(u'fork.13780').connect(): child process stdin/stdout=17
[pid 53505] 17:20:27.531426 D mitogen: Adding route to 3 via mitogen.ssh.Stream(u'ssh.192.168.33.12')
[pid 53505] 17:20:27.532111 D mitogen: Router(Broker(0x11191ea90)).add_route(3, mitogen.ssh.Stream(u'ssh.192.168.33.12'))
[pid 53505] 17:20:27.533233 D mitogen.ctx.ssh.192.168.33.12: mitogen: register(Context(3, u'fork.13780'), mitogen.fork.Stream(u'fork.13780'))
[pid 53505] 17:20:27.534052 D mitogen.ctx.ssh.192.168.33.12: ansible_mitogen.target: Selected temp directory: u'/tmp/.ansible-vagrant/tmp' (from [u'/tmp/.ansible-vagrant/tmp', u'/var/tmp', u'/tmp', '/tmp', '/var/tmp', '/usr/tmp', '/home/vagrant'])
[pid 53505] 17:20:27.534873 D mitogen.ctx.ssh.192.168.33.12: mitogen: _dispatch_calls: Message(2, 0, 0, 101, 1000, '\x80\x02(NX\x16\x00\x00\x00ansible_mitogen.targetNX\n\x00\x00\x00init_child)cm'..185) -> {u'fork_context': Context(3, u'fork.13780'), u'home_dir': u'/home/vagrant', u'good_temp_dir': u'/tmp/.ansible-vagrant/tmp'}
[pid 53505] 17:20:27.538191 D mitogen.ctx.fork.13780: mitogen: register(Context(2, 'parent'), mitogen.core.Stream('parent'))
[pid 53505] 17:20:27.540185 D mitogen: CallChain(Context(2, u'ssh.192.168.33.12')).call_async(): mitogen.parent._proxy_connect(method_name=u'su', name=None, kwargs=Kwargs({u'username': u'db2inst1', u'profiling': False, u'python_path': [u'/usr/bin/python'], 'unidirectional': True, u'debug': True, u'su_path': None, u'password': [secret], u'connect_timeout': 10}))
[pid 53505] 17:20:27.541204 D mitogen.ctx.fork.13780: mitogen: Connected to Context(2, 'parent'); my ID is 3, PID is 13780
[pid 53505] 17:20:27.542258 D mitogen: Context(2, u'ssh.192.168.33.12').send_async(Message(2, 0, 0, 101, 1001, '\x80\x02(NX\x0e\x00\x00\x00mitogen.parentNX\x0e\x00\x00\x00_proxy_connect)cmitog'..351))
[pid 53505] 17:20:27.543250 D mitogen.ctx.fork.13780: mitogen: Recovered sys.executable: '/usr/bin/python'
[pid 53505] 17:20:27.561552 D mitogen.ctx.ssh.192.168.33.12: mitogen: _dispatch_one((None, u'mitogen.parent', None, u'_proxy_connect', (), Kwargs({'method_name': u'su', 'name': None, 'kwargs': Kwargs({u'username': u'db2inst1', u'profiling': False, u'python_path': [u'/usr/bin/python'], 'unidirectional': True, u'debug': True, u'su_path': None, u'password': [secret], u'connect_timeout': 10})})))
[pid 53505] 17:20:27.564731 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('mitogen.su')
[pid 53505] 17:20:27.565780 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer.load_module(u'mitogen.su')
[pid 53505] 17:20:27.567847 D mitogen.ctx.ssh.192.168.33.12: mitogen: _request_module(u'mitogen.su'): new request
[pid 53505] 17:20:27.568717 D mitogen: ModuleResponder(Router(Broker(0x11191ea90)))._on_get_module('mitogen.su')
[pid 53505] 17:20:27.571298 D mitogen: _send_load_module(mitogen.ssh.Stream(u'ssh.192.168.33.12'), u'mitogen.su')
[pid 53505] 17:20:27.581961 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer._on_load_module(u'mitogen.su')
[pid 53505] 17:20:27.588042 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.su.Stream(u'default').connect()
[pid 53505] 17:20:27.616119 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('encodings.base64_codec')
[pid 53505] 17:20:27.617965 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer(): 'encodings.base64_codec' is submodule of a package we did not load
[pid 53505] 17:20:27.619217 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('encodings.base64')
[pid 53505] 17:20:27.620217 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer(): 'encodings.base64' is submodule of a package we did not load
[pid 53505] 17:20:27.621565 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('base64')
[pid 53505] 17:20:27.623912 D mitogen.ctx.ssh.192.168.33.12: mitogen: tty_create_child() child 13782 fd 18, parent 13771, cmd: su db2inst1 -c "/usr/bin/python -c \"import codecs,os,sys;_=codecs.decode;exec(_(_(\\\"eNqFkTFrwzAQhef4V3iTRIQjxUPAIGjJUDqUginN0IYi23IiqkhCduKmv74XuxA7Hbrdx3un9zjldCNck3jtFSZRoN2IdB0D1C58YpJFM5iro19iRjlj5Mo5HVMAlQ9cGtconI8hjGEzhg4AApszxBvZQuohFiJGlQydtiiWtupF9aXKYysLo3p5cWzCotB24c/t3lkEPWc3trnoF08qNNrZtyzd9rHKnnQARvf5wytDWzFdGzyABk8FOsU5wgfdup2y2UnugrTtnXWVYjwJe2US40ppMp6uVpwgEsGbXdCtwpyip8eXZ8bYu0XQpYQdTEi0Fh/4cvTKeWXh1CgUiCRByQrz5SpNCUXf2sNLtRdX34airkCXf6j9b8C6n4fb3ri7/9x/W/JJyx/uq7Gg\\\".encode(),\\\"base64\\\"),\\\"zip\\\"))\""
[pid 53505] 17:20:27.625074 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.su.Stream(u'local.13782').connect(): child process stdin/stdout=18
[pid 53505] 17:20:28.965628 D mitogen.ctx.ssh.192.168.33.12: mitogen.su: mitogen.su.Stream(u'local.13782'): received '\xd0\x9f\xd0\xb0\xd1\x80\xd0\xbe\xd0\xbb\xd1\x8c: '
[pid 53505] 17:20:37.587632 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.su.Stream(u'local.13782'): child process still alive, sending SIGTERM
[pid 53505] 17:20:37.588485 D mitogen.ctx.ssh.192.168.33.12: mitogen: _dispatch_calls: Message(2, 0, 0, 101, 1001, '\x80\x02(NX\x0e\x00\x00\x00mitogen.parentNX\x0e\x00\x00\x00_proxy_connect)cmitog'..351) -> {u'msg': 'error occurred on host node01.rhel.local: Connection timed out.', u'id': None, u'name': None}
[pid 53506] 17:20:37.597285 D mitogen: mitogen.core.Stream(u'unix_listener.53505').on_disconnect()
[pid 53506] 17:20:37.598276 D mitogen: Waker(Broker(0x11196ff90) rfd=45, wfd=46).on_disconnect()
[pid 53505] 17:20:37.598553 D mitogen: mitogen.core.Stream(u'unix_client.53506').on_disconnect()
[pid 53505] 17:20:37.601531 D mitogen: Context(1, None).on_disconnect()
fatal: [node01]: FAILED! => {}
MSG:
error occurred on host node01.rhel.local: Connection timed out.
Hello. I guess i found a bag using a privilege escalation in an ansible playbook. I reduced the playbook to describe a behavior.
When I play it I get an error in task "command: "whoami""
error occurred on host DB2TST_N1.vdatacenter.it.ru: EOF on stream; last 300 bytes received: u'3004-501 Cannot su to "db2inst1" : Authentication is denied.\n'
The second task completes normally. Logs from target: TASK aix_db2.txtOk, I solve the problem replacing "become: yes" to task
In this way all works fine. But in real project I have many huge playbooks which 99% consist of task needs "become: yes". And only 1% tasks needs constructions like this
become_user: "db2inst1" become_method: "su" become_flags: "-"