dw
commented
5 years ago This looks like it should be a simple fix -- however I am worried about AIX target :) None of this code is tested on AIX
Open diioc opened 5 years ago
dw
commented
5 years ago This looks like it should be a simple fix -- however I am worried about AIX target :) None of this code is tested on AIX
dw
commented
5 years ago I notice some issues:
If you log into the system, does cutpasting that command line produce a password prompt? su db2inst1 -c "/usr/bin/python -c \"print 123\""
If it does not, is it possible to check the system log? AIX is rejecting the authentication for some reason -- it will not even ask for a password. This suggests something about Mitogen's process environment is incorrect, and as a result, something like PAM is refusing to do business
dw
commented
5 years ago It is possible your missing 'become_flags' is causing the problem, but at least for now I doubt it.
dw
commented
5 years ago From googling "authentication is denied", it may relate to AIX RBAC authentication
diioc
commented
5 years ago I tried to execute su db2inst1 -c "/usr/bin/python -c \"print 123\"" in multiple different ways:
sudo su db2inst1 -c "/usr/bin/python -c \"print 123\"" => prompts for a sudo passI'll try to reproduce it on rhel7
dw
commented
5 years ago If it works when you are logged in interactively, then it must be some problem with how Mitogen creates processes or TTYs, or perhaps because Mitogen launches a non-interactive SSH session, it does not receive some important privilege.
I'm afraid I do not know enough about AIX to understand this issue :) Do you know of an expert within your organization that would understand deeper security issues? I am fairly sure this is some strange security restriction AIX has, and to make things work, we must meet those restriction.
diioc
commented
5 years ago Hi man! Guess what?) I reproduced the issue on rhel7. It has a different output in same situation but I can fix it by the same way to replace 'become: yes' statement.
It don't work
It works fine
Logs from node mitogen.13780.20190212_172029.log mitogen.13771.20190212_172028.log
And output from host:
ansible-playbook mit_test.yml -i inventory -vvv --limit node01
ansible-playbook 2.6.1
config file = /Users/diioc/devops/vagrant/rhel-db2/provisioning/ansible.cfg
configured module search path = [u'/Users/diioc/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
executable location = /usr/local/bin/ansible-playbook
python version = 2.7.15 (default, Jun 17 2018, 12:46:58) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
Using /Users/diioc/devops/vagrant/rhel-db2/provisioning/ansible.cfg as config file
Parsed /Users/diioc/devops/vagrant/rhel-db2/provisioning/inventory inventory source with ini plugin
PLAYBOOK: mit_test.yml **********************************************************************************
1 plays in mit_test.yml
PLAY [node] *********************************************************************************************
[pid 53505] 17:20:25.865219 D mitogen: mitogen.service.Pool(0x1119483d0, size=16, th='MainThread'): initialized
[pid 53505] 17:20:25.870836 D ansible_mitogen.process: Service pool configured: size=16
META: ran handlers
TASK [command] ******************************************************************************************
task path: /Users/diioc/devops/vagrant/rhel-db2/provisioning/mit_test.yml:10
[pid 53506] 17:20:25.916618 D mitogen: unix.connect(path='/var/folders/pf/rc9rjb7x2_1d3r_73cq5l0w80000gn/T/mitogen_unix_wPEhwd')
[pid 53506] 17:20:25.918792 D mitogen: unix.connect(): local ID is 1, remote is 0
[pid 53505] 17:20:25.918860 D mitogen: register(Context(1, None), mitogen.core.Stream(u'unix_client.53506'))
[pid 53505] 17:20:25.928756 D mitogen: mitogen.ssh.Stream(u'default').connect()
[pid 53505] 17:20:26.096152 D mitogen: create_child() child 53507 fd 95, parent 53505, cmd: ssh -o "LogLevel ERROR" -l vagrant -o "Compression yes" -o "ServerAliveInterval 15" -o "ServerAliveCountMax 3" -o "BatchMode yes" -o "StrictHostKeyChecking yes" -C -o ControlMaster=auto -o ControlPersist=60s 192.168.33.12 /usr/bin/python -c "'import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqFkVFLwzAUhZ/XX9G3JCxuycYQCgFlD+KDCEXsgw5Jm1SDWRLSbnX+eu86Ye188O1+3HNzDic5LYRvZsEEjUkSaTcgU6cAtY+fmGTJBGa1CwvMKGeMnDmnQ4qw5SeurG80zocQh1AMoQMAw+YA9la24LpNhUiRkrEzDqXSqX6pv3S1a2Vpdb+e75o4L42bh0P74R2CnJML2VT0h3sdG+PdS7bc9Lba7U0ERrf53TNDGzE+O2kALR4v6BinCG9N69+1y5QxvrqRjbcHZ9yV2cpqZn0lbbZartiKIJLAm100rcacoof7p0fG2KtDkKXyCionyVq84WPpygftoGoUS0RmUUuF+eKac0LRtwnwUh3EWVdQ1JXo+A91+DVY9/Op2wt195/6b0o+SvkDRZeyBA==\".encode(),\"base64\"),\"zip\"))'"
[pid 53505] 17:20:26.098825 D mitogen: mitogen.ssh.Stream(u'local.53507').connect(): child process stdin/stdout=95
[pid 53505] 17:20:26.508275 D mitogen: mitogen.ssh.Stream(u'local.53507'): received 'MITO000\n'
[pid 53505] 17:20:26.509444 D mitogen: mitogen.ssh.Stream(u'local.53507')._ec0_received()
[pid 53505] 17:20:26.525306 D mitogen: register(Context(2, u'ssh.192.168.33.12'), mitogen.ssh.Stream(u'ssh.192.168.33.12'))
[pid 53505] 17:20:26.530016 D mitogen: CallChain(Context(2, u'ssh.192.168.33.12')).call_async(): ansible_mitogen.target.init_child(candidate_temp_dirs=[u'/tmp/.ansible-${USER}/tmp', u'/var/tmp', u'/tmp'], log_level=10)
[pid 53505] 17:20:26.531752 D mitogen: Context(2, u'ssh.192.168.33.12').send_async(Message(2, 0, 0, 101, 1000, '\x80\x02(NX\x16\x00\x00\x00ansible_mitogen.targetNX\n\x00\x00\x00init_child)cm'..185))
[pid 53505] 17:20:26.540609 D mitogen: _build_tuple('/usr/local/lib/python2.7/site-packages/ansible/__init__.py', u'ansible') -> [u'cli', u'compat', u'config', u'constants', u'errors', u'executor', u'galaxy', u'inventory', u'module_utils', u'modules', u'parsing', u'playbook', u'plugins', u'release', u'template', u'utils', u'vars']
[pid 53505] 17:20:26.542824 D mitogen: _send_load_module(mitogen.ssh.Stream(u'ssh.192.168.33.12'), u'ansible.release')
[pid 53505] 17:20:26.544781 D mitogen: _send_load_module(mitogen.ssh.Stream(u'ssh.192.168.33.12'), u'ansible')
[pid 53505] 17:20:26.550907 D mitogen: _build_tuple('/usr/local/lib/python2.7/site-packages/ansible/module_utils/__init__.py', u'ansible.module_utils') -> [u'_text', u'acme', u'ansible_release', u'ansible_tower', u'api', u'aws', u'azure_rm_common', u'azure_rm_common_rest', u'basic', u'cloud', u'cloudscale', u'cloudstack', u'common', u'connection', u'crypto', u'database', u'digital_ocean', u'dimensiondata', u'docker_common', u'ec2', u'exoscale', u'f5_utils', u'facts', u'firewalld', u'gcdns', u'gce', u'gcp', u'gcp_utils', u'gitlab', u'heroku', u'infinibox', u'influxdb', u'ipa', u'ismount', u'json_utils', u'k8s', u'keycloak', u'known_hosts', u'ldap', u'lxd', u'manageiq', u'memset', u'mysql', u'net_tools', u'netapp', u'network', u'oneandone', u'oneview', u'opennebula', u'openstack', u'ovirt', u'parsing', u'postgres', u'powershell', u'pure', u'pycompat24', u'rax', u'redhat', u'remote_management', u'scaleway', u'service', u'six', u'splitter', u'univention_umc', u'urls', u'vca', u'vmware', u'vmware_rest_client', u'vultr']
...
...
Importer().find_module('ansible.module_utils.parsing')
[pid 53505] 17:20:27.499786 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer.load_module(u'ansible.module_utils.parsing')
[pid 53505] 17:20:27.500749 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('ansible.module_utils.parsing.convert_bool')
[pid 53505] 17:20:27.501596 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer.load_module(u'ansible.module_utils.parsing.convert_bool')
[pid 53505] 17:20:27.502350 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('ansible.module_utils.parsing.ansible')
[pid 53505] 17:20:27.503102 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer(): master doesn't know 'ansible.module_utils.parsing.ansible'
[pid 53505] 17:20:27.503874 D mitogen.ctx.ssh.192.168.33.12: ansible_mitogen.runner: EnvironmentFileWatcher(u'/home/vagrant/.pam_environment') installed; existing keys: []
[pid 53505] 17:20:27.504614 D mitogen.ctx.ssh.192.168.33.12: ansible_mitogen.runner: EnvironmentFileWatcher(u'/etc/environment') installed; existing keys: []
[pid 53505] 17:20:27.505436 D mitogen.ctx.ssh.192.168.33.12: mitogen: replaced Poller(0x7f32066a35d0) with EpollPoller(0x7f31faddf910) (new: 4 readers, 1 writers; old: 4 readers, 1 writers)
[pid 53505] 17:20:27.506212 D mitogen.ctx.ssh.192.168.33.12: mitogen: Router(Broker(0x7f32066a3490)).upgrade()
[pid 53505] 17:20:27.507037 D mitogen: IdAllocator(Router(Broker(0x11191ea90))): allocating [3..1003)
[pid 53505] 17:20:27.507593 D mitogen: IdAllocator(Router(Broker(0x11191ea90))): allocating [3..1003) to Context(2, u'ssh.192.168.33.12')
[pid 53505] 17:20:27.529339 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.fork.Stream(u'default').connect()
[pid 53505] 17:20:27.530358 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.fork.Stream(u'fork.13780').connect(): child process stdin/stdout=17
[pid 53505] 17:20:27.531426 D mitogen: Adding route to 3 via mitogen.ssh.Stream(u'ssh.192.168.33.12')
[pid 53505] 17:20:27.532111 D mitogen: Router(Broker(0x11191ea90)).add_route(3, mitogen.ssh.Stream(u'ssh.192.168.33.12'))
[pid 53505] 17:20:27.533233 D mitogen.ctx.ssh.192.168.33.12: mitogen: register(Context(3, u'fork.13780'), mitogen.fork.Stream(u'fork.13780'))
[pid 53505] 17:20:27.534052 D mitogen.ctx.ssh.192.168.33.12: ansible_mitogen.target: Selected temp directory: u'/tmp/.ansible-vagrant/tmp' (from [u'/tmp/.ansible-vagrant/tmp', u'/var/tmp', u'/tmp', '/tmp', '/var/tmp', '/usr/tmp', '/home/vagrant'])
[pid 53505] 17:20:27.534873 D mitogen.ctx.ssh.192.168.33.12: mitogen: _dispatch_calls: Message(2, 0, 0, 101, 1000, '\x80\x02(NX\x16\x00\x00\x00ansible_mitogen.targetNX\n\x00\x00\x00init_child)cm'..185) -> {u'fork_context': Context(3, u'fork.13780'), u'home_dir': u'/home/vagrant', u'good_temp_dir': u'/tmp/.ansible-vagrant/tmp'}
[pid 53505] 17:20:27.538191 D mitogen.ctx.fork.13780: mitogen: register(Context(2, 'parent'), mitogen.core.Stream('parent'))
[pid 53505] 17:20:27.540185 D mitogen: CallChain(Context(2, u'ssh.192.168.33.12')).call_async(): mitogen.parent._proxy_connect(method_name=u'su', name=None, kwargs=Kwargs({u'username': u'db2inst1', u'profiling': False, u'python_path': [u'/usr/bin/python'], 'unidirectional': True, u'debug': True, u'su_path': None, u'password': [secret], u'connect_timeout': 10}))
[pid 53505] 17:20:27.541204 D mitogen.ctx.fork.13780: mitogen: Connected to Context(2, 'parent'); my ID is 3, PID is 13780
[pid 53505] 17:20:27.542258 D mitogen: Context(2, u'ssh.192.168.33.12').send_async(Message(2, 0, 0, 101, 1001, '\x80\x02(NX\x0e\x00\x00\x00mitogen.parentNX\x0e\x00\x00\x00_proxy_connect)cmitog'..351))
[pid 53505] 17:20:27.543250 D mitogen.ctx.fork.13780: mitogen: Recovered sys.executable: '/usr/bin/python'
[pid 53505] 17:20:27.561552 D mitogen.ctx.ssh.192.168.33.12: mitogen: _dispatch_one((None, u'mitogen.parent', None, u'_proxy_connect', (), Kwargs({'method_name': u'su', 'name': None, 'kwargs': Kwargs({u'username': u'db2inst1', u'profiling': False, u'python_path': [u'/usr/bin/python'], 'unidirectional': True, u'debug': True, u'su_path': None, u'password': [secret], u'connect_timeout': 10})})))
[pid 53505] 17:20:27.564731 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('mitogen.su')
[pid 53505] 17:20:27.565780 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer.load_module(u'mitogen.su')
[pid 53505] 17:20:27.567847 D mitogen.ctx.ssh.192.168.33.12: mitogen: _request_module(u'mitogen.su'): new request
[pid 53505] 17:20:27.568717 D mitogen: ModuleResponder(Router(Broker(0x11191ea90)))._on_get_module('mitogen.su')
[pid 53505] 17:20:27.571298 D mitogen: _send_load_module(mitogen.ssh.Stream(u'ssh.192.168.33.12'), u'mitogen.su')
[pid 53505] 17:20:27.581961 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer._on_load_module(u'mitogen.su')
[pid 53505] 17:20:27.588042 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.su.Stream(u'default').connect()
[pid 53505] 17:20:27.616119 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('encodings.base64_codec')
[pid 53505] 17:20:27.617965 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer(): 'encodings.base64_codec' is submodule of a package we did not load
[pid 53505] 17:20:27.619217 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('encodings.base64')
[pid 53505] 17:20:27.620217 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer(): 'encodings.base64' is submodule of a package we did not load
[pid 53505] 17:20:27.621565 D mitogen.ctx.ssh.192.168.33.12: mitogen: Importer().find_module('base64')
[pid 53505] 17:20:27.623912 D mitogen.ctx.ssh.192.168.33.12: mitogen: tty_create_child() child 13782 fd 18, parent 13771, cmd: su db2inst1 -c "/usr/bin/python -c \"import codecs,os,sys;_=codecs.decode;exec(_(_(\\\"eNqFkTFrwzAQhef4V3iTRIQjxUPAIGjJUDqUginN0IYi23IiqkhCduKmv74XuxA7Hbrdx3un9zjldCNck3jtFSZRoN2IdB0D1C58YpJFM5iro19iRjlj5Mo5HVMAlQ9cGtconI8hjGEzhg4AApszxBvZQuohFiJGlQydtiiWtupF9aXKYysLo3p5cWzCotB24c/t3lkEPWc3trnoF08qNNrZtyzd9rHKnnQARvf5wytDWzFdGzyABk8FOsU5wgfdup2y2UnugrTtnXWVYjwJe2US40ppMp6uVpwgEsGbXdCtwpyip8eXZ8bYu0XQpYQdTEi0Fh/4cvTKeWXh1CgUiCRByQrz5SpNCUXf2sNLtRdX34airkCXf6j9b8C6n4fb3ri7/9x/W/JJyx/uq7Gg\\\".encode(),\\\"base64\\\"),\\\"zip\\\"))\""
[pid 53505] 17:20:27.625074 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.su.Stream(u'local.13782').connect(): child process stdin/stdout=18
[pid 53505] 17:20:28.965628 D mitogen.ctx.ssh.192.168.33.12: mitogen.su: mitogen.su.Stream(u'local.13782'): received '\xd0\x9f\xd0\xb0\xd1\x80\xd0\xbe\xd0\xbb\xd1\x8c: '
[pid 53505] 17:20:37.587632 D mitogen.ctx.ssh.192.168.33.12: mitogen: mitogen.su.Stream(u'local.13782'): child process still alive, sending SIGTERM
[pid 53505] 17:20:37.588485 D mitogen.ctx.ssh.192.168.33.12: mitogen: _dispatch_calls: Message(2, 0, 0, 101, 1001, '\x80\x02(NX\x0e\x00\x00\x00mitogen.parentNX\x0e\x00\x00\x00_proxy_connect)cmitog'..351) -> {u'msg': 'error occurred on host node01.rhel.local: Connection timed out.', u'id': None, u'name': None}
[pid 53506] 17:20:37.597285 D mitogen: mitogen.core.Stream(u'unix_listener.53505').on_disconnect()
[pid 53506] 17:20:37.598276 D mitogen: Waker(Broker(0x11196ff90) rfd=45, wfd=46).on_disconnect()
[pid 53505] 17:20:37.598553 D mitogen: mitogen.core.Stream(u'unix_client.53506').on_disconnect()
[pid 53505] 17:20:37.601531 D mitogen: Context(1, None).on_disconnect()
fatal: [node01]: FAILED! => {}
MSG:
error occurred on host node01.rhel.local: Connection timed out.
Hello. I guess i found a bag using a privilege escalation in an ansible playbook. I reduced the playbook to describe a behavior.
When I play it I get an error in task "command: "whoami""
error occurred on host DB2TST_N1.vdatacenter.it.ru: EOF on stream; last 300 bytes received: u'3004-501 Cannot su to "db2inst1" : Authentication is denied.\n'The second task completes normally. Logs from target: TASK aix_db2.txtOk, I solve the problem replacing "become: yes" to task
In this way all works fine. But in real project I have many huge playbooks which 99% consist of task needs "become: yes". And only 1% tasks needs constructions like this
become_user: "db2inst1" become_method: "su" become_flags: "-"