Mitogen 2.8 tmp noexec - Githubissues

bigHosting commented 5 years ago

After installing and configuring mitogen plugin, hardened hosts fail. I was able to trace the issue to /tmp being confgured with noexec option. In my hardened environment, noexec on /tmp is required. Is there any way I can tell mitogen to use a different folder for tmp files ?

Which version of Ansible are you running? 2.8.3
Is your version of Ansible patched in any way? No
Are you running with any custom modules, or module_utils loaded? No
Have you tried the latest master version from Git? No
Do you have some idea of what the underlying problem may be? Yes, /tmp with noexec in /etc/fstab will make mitogen fail
Mention your host and target OS and versions CentOS 7 latest release, patched up2date
Mention your host and target Python versions python 2.7

Error:

TASK [Gathering Facts] ***** Sunday 18 August 2019 21:22:42 -0400 (0:00:00.276) 0:00:00.276 * An exception occurred during task execution. To see the full traceback, use -vvv. The error was: 'paths': '\n '.join(paths), fatal: [scan1.sec.hostopia.com]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}

[mux 15813] 21:30:28.605469 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.common.file [mux 15813] 21:30:28.608249 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.common.sys_info [mux 15813] 21:30:28.608458 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.distro [mux 15813] 21:30:28.608647 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting distro [mux 15813] 21:30:28.608808 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.distro._distro [mux 15813] 21:30:28.616025 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule os [mux 15813] 21:30:28.616355 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule re [mux 15813] 21:30:28.616990 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule sys [mux 15813] 21:30:28.617163 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule json [mux 15813] 21:30:28.617323 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule shlex [mux 15813] 21:30:28.617483 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule logging [mux 15813] 21:30:28.617643 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule optparse [mux 15813] 21:30:28.617802 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting org [mux 15813] 21:30:28.619699 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.distro has no submodule subprocess [mux 15813] 21:30:28.621779 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.common._utils [mux 15813] 21:30:28.621996 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.pycompat24 [mux 15813] 21:30:28.623364 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils has no submodule sys [mux 15813] 21:30:28.624732 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils has no submodule ast [mux 15813] 21:30:28.624922 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.common.parameters [mux 15813] 21:30:28.626459 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.common.collections [mux 15813] 21:30:28.626641 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.common.validation [mux 15813] 21:30:28.632432 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.parsing [mux 15813] 21:30:28.632640 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: requesting ansible.module_utils.parsing.convert_bool [mux 15813] 21:30:28.632830 D mitogen.importer.[ssh.scan1.sec.hostopia.com]: ansible.module_utils.parsing has no submodule ansible [mux 15813] 21:30:28.633032 D ansible_mitogen.runner.[ssh.scan1.sec.hostopia.com]: EnvironmentFileWatcher('/home/hdeploy/.pam_environment') installed; existing keys: [] [mux 15813] 21:30:28.633778 D ansible_mitogen.runner.[ssh.scan1.sec.hostopia.com]: EnvironmentFileWatcher('/etc/environment') installed; existing keys: [] [mux 15813] 21:30:28.636166 D mitogen.parent.[ssh.scan1.sec.hostopia.com]: upgraded Poller with EpollPoller (new: 4 readers, 0 writers; old: 4 readers, 0 writers) [mux 15813] 21:30:28.636336 D mitogen.parent.[ssh.scan1.sec.hostopia.com]: upgrading Router(Broker(a510)) with capabilities to start new children [mux 15813] 21:30:28.636500 D mitogen.[ssh.scan1.sec.hostopia.com]: sending message to Context(0, u'master'): Message(0, 2, 2, 105, 1000, ''..0) [mux 15813] 21:30:28.636673 D mitogen: IdAllocator(Router(Broker(4510))): allocating [3..1003) [mux 15813] 21:30:28.636822 D mitogen: IdAllocator(Router(Broker(4510))): allocating [3..1003) to Context(2, u'ssh.scan1.sec.hostopia.com') [mux 15813] 21:30:28.639312 D mitogen.parent.[ssh.scan1.sec.hostopia.com]: creating connection to context 3 using mitogen.fork [mux 15813] 21:30:28.643436 D mitogen.parent.[ssh.scan1.sec.hostopia.com]: child for Connection(None) started: pid:25825 stdin:14 stdout:14 stderr:None [mux 15813] 21:30:28.643618 D mitogen.parent.[ssh.scan1.sec.hostopia.com]: BootstrapProtocol(fork.25825): new child booted successfully [mux 15813] 21:30:28.643780 D mitogen.[ssh.scan1.sec.hostopia.com]: Router(Broker(a510)): registering Context(3, u'fork.25825') to stream <Stream fork.25825 #7310> [mux 15813] 21:30:28.643985 D mitogen.route_monitor: Adding route to 3 via <Stream ssh.scan1.sec.hostopia.com #0fd0> [mux 15813] 21:30:28.644128 D mitogen.parent: Router(Broker(4510)): adding route to context 3 via <Stream ssh.scan1.sec.hostopia.com #0fd0> [mux 15813] 21:30:28.646511 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir u'/home/hdeploy/.ansible/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.646718 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir u'/var/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.646893 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir u'/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.647056 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir '/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.646893 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir u'/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.647056 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir '/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.647245 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir '/var/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.647966 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir '/usr/tmp' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.648162 D mitogen.[fork.25825]: Router(Broker(ab10)): registering Context(2, u'parent') to stream <Stream parent #7750> [mux 15813] 21:30:28.648324 D mitogen.[fork.25825]: Python version is 2.7.5 (default, Jun 20 2019, 20:27:34) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] [mux 15813] 21:30:28.648513 D mitogen.[fork.25825]: Parent is context 2 (parent); my ID is 3 [mux 15813] 21:30:28.648673 D mitogen.[fork.25825]: pid:25825 ppid:25809 uid:400/400, gid:400/400 host:'scan1.sec.hostopia.com' [mux 15813] 21:30:28.648868 D mitogen.[fork.25825]: Recovered sys.executable: '/usr/bin/python' [mux 15813] 21:30:28.649619 D ansible_mitogen.target.[ssh.scan1.sec.hostopia.com]: temp dir '/home/hdeploy' unusable: filesystem appears to be mounted noexec [mux 15813] 21:30:28.649831 D mitogen.[ssh.scan1.sec.hostopia.com]: Dispatcher: Message(2, 0, 0, 101, 1000, '\x80\x02(NX\x16\x00\x00\x00ansible_mitogen.targetNX\n\x00\x00\x00init_child)cm'..174) -> CallError(u'exceptions.IOError: Unable to find a useable temporary directory. This likely means no\nsystem-supplied TMP directory can be written to, or all directories\nwere mounted on \'noexec\' filesystems.\n\nThe following paths were tried:\n /home/hdeploy/.ansible/tmp\n /var/tmp\n /tmp\n /tmp\n /var/tmp\n /usr/tmp\n /home/hdeploy\n\nPlease check \'-vvv\' output for a log of individual path errors.\n File "", line 3661, in _dispatch_one\n File "master:/usr/lib/python2.7/site-packages/ansible_mitogen/target.py", line 383, in init_child\n good_temp_dir = find_good_temp_dir(candidate_temp_dirs)\n File "master:/usr/lib/python2.7/site-packages/ansible_mitogen/target.py", line 331, in find_good_temp_dir\n \'paths\': \'\n \'.join(paths),\n',) [task 15847] 21:30:28.655236 D mitogen: MitogenProtocol(unix_listener.15813): disconnecting [task 15847] 21:30:28.655806 D mitogen: Waker(fd=11/12): disconnecting [mux 15813] 21:30:28.655876 D mitogen: <Side of unix_client.15847 fd 76>: empty read, disconnecting [mux 15813] 21:30:28.656062 D mitogen: MitogenProtocol(unix_client.15847): disconnecting [task 15847] 21:30:28.656148 D mitogen: Router(Broker(8b50)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg. The full traceback is: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 145, in run res = self._execute() File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 650, in _execute result = self._handler.run(task_vars=variables) File "/usr/lib/python2.7/site-packages/ansible_mitogen/mixins.py", line 116, in run return super(ActionModuleMixin, self).run(tmp, task_vars) File "/usr/lib/python2.7/site-packages/ansible/plugins/action/gather_facts.py", line 66, in run res = self._execute_module(module_name=fact_module, module_args=mod_args, task_vars=task_vars, wrap_async=False) File "/usr/lib/python2.7/site-packages/ansible_mitogen/mixins.py", line 346, in _execute_module self._temp_file_gibberish(module_args, wrap_async) File "/usr/lib/python2.7/site-packages/ansible_mitogen/mixins.py", line 325, in _temp_file_gibberish self._connection.get_good_temp_dir() File "/usr/lib/python2.7/site-packages/ansible_mitogen/connection.py", line 762, in get_good_temp_dir self._connect() File "/usr/lib/python2.7/site-packages/ansible_mitogen/connection.py", line 782, in _connect self._connect_stack(stack) File "/usr/lib/python2.7/site-packages/ansible_mitogen/connection.py", line 735, in _connect_stack stack=mitogen.utils.cast(list(stack)), File "/usr/lib/python2.7/site-packages/mitogen/service.py", line 120, in call return call_context.call_service(service_name, method_name, **kwargs) File "/usr/lib/python2.7/site-packages/mitogen/core.py", line 2271, in call_service return recv.get().unpickle() File "/usr/lib/python2.7/site-packages/mitogen/core.py", line 963, in unpickle raise obj CallError: exceptions.IOError: Unable to find a useable temporary directory. This likely means no system-supplied TMP directory can be written to, or all directories were mounted on 'noexec' filesystems.

The following paths were tried: /home/hdeploy/.ansible/tmp /var/tmp /tmp /tmp /var/tmp /usr/tmp /home/hdeploy

Please check '-vvv' output for a log of individual path errors. File "", line 3661, in _dispatch_one File "master:/usr/lib/python2.7/site-packages/ansible_mitogen/target.py", line 383, in init_child good_temp_dir = find_good_temp_dir(candidate_temp_dirs) File "master:/usr/lib/python2.7/site-packages/ansible_mitogen/target.py", line 331, in find_good_temp_dir 'paths': '\n '.join(paths),

fatal: [scan1.sec.hostopia.com]: FAILED! => { "msg": "Unexpected failure during module execution.", "stdout": "" }

dw commented 5 years ago

This might be a new regression in 2.8. Was it working for you in any previous version?

The difference here compared to Ansible is that I don't think Ansible bothers checking if the temp directory is noexec or not, but we do, for some reason that's bitten quite recently (#575).

I cannot find why that check was added, but I'm of the conviction it was added for good reason. Not least -- custom binary modules won't run if they pick the wrong filesystem.

I think this can be considered another case of what's described in the comments of #575. As in that ticket, it seems the only option here is to remove the noexec check and find out what breaks :)

Thanks for reporting this! I will look at removing the check tomorrow, but in the meantime, you can replace the line:

 if not os.access(tmp.name, os.X_OK):

In ansible_mitogen/target.py with the line:

if False:

To see if it gets you moving again.

dw commented 5 years ago

The plugin should be respecting the ANSIBLE_REMOTE_TMP variable documented here: https://docs.ansible.com/ansible/2.5/plugins/shell/sh.html

bigHosting commented 5 years ago

Hi David,

I was getting this error in an older version of mitogen. I tried the above suggestion but it didn't seem not to get the expected results.

304 try: 305 # access(.., X_OK) is sufficient to detect noexec. 306 #if not os.access(tmp.name, os.X_OK): 307 if False: 308 raise OSError('filesystem appears to be mounted noexec') 309 except OSError:

However, since mitogen plugin is looking for /usr/tmp, /usr should always have the exec flag on, so the below worked on CentOS 7:

root@syslog2n2[/usr]# rm -f /usr/tmp root@syslog2n2[/usr]# mkdir -m 1773 /usr/tmp

Thank you a lot for the ANSIBLE_REMOTE_TMP, I'll try using it with a custom folder so I don't have to modify OS default /usr/tmp ( symlink to /var/tmp ) . Mitogen is making life so much easier !

dw commented 5 years ago

You should not have to modify anything :) The fact it deviates from Ansible behaviour is already a bug. Temp file handling is a piece of extremely dirty laundry in Ansible, Mitogen is forced to play the same game to remain compatible. This is just another case it needs to handle

Will take another look at this tomorrow. Thanks again for reporting

Ant1B2x commented 2 years ago

Hi @dw. Do you have any news on this issue? It seems like the problem is still a thing. The only solutions I found were either to comment the check as you suggested, or to use a raw task in Ansible playbook to assure tmp directory permissions are 777 (maybe you could indicate it on your website?).

Anyway thanks for your awesome work on Mitogen and have a nice day!

mitogen-hq / mitogen

Mitogen 2.8 tmp noexec #632