Copy module does not cleanup temp files on failure

SUMMARY

When using the copy module, if the destination file system does not have enough space the copy fails but the temp file deployed to the server via the copy action plugin remains on the system consuming space.

This appears to be happening in the atomic_move method in module_utils/basic.py.

ISSUE TYPE

Bug Report

COMPONENT NAME

copy

ANSIBLE VERSION

ansible 2.8.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/Users/jowestco/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jowestco/Library/Python/venvs/ansible-latest/lib/python3.7/site-packages/ansible
  executable location = /Users/jowestco/Library/Python/venvs/ansible-latest/bin/ansible
  python version = 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) [Clang 6.0 (clang-600.0.57)]

CONFIGURATION

RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False
RETRY_FILES_SAVE_PATH(/etc/ansible/ansible.cfg) = /tmp

OS / ENVIRONMENT

Red Hat Enterprise Linux release 8.0 (Ootpa) There are two partitions on this machine: /dev/mapper/rhel-root 6.2G 2.0G 4.3G 32% / /dev/sdb1 1017M 1017M 196K 100% /python_scripts

/root/.ansible is on the / partition. The /python_scripts is the destination directory for the file copy.

STEPS TO REPRODUCE

Try to copy a file that is 200k to the /python_scripts directory. The copy action plugin will be able to copy file out to /root/.ansible/tmp/ but the copy module will not be able to move that file to /python_scripts because there is not enough free space. After the playbook ends you can see the left over temp files in /root/.ansible/tmp/.

---
- name: Create a copy failure
  hosts: remotes
  gather_facts: False
  tasks:
    - copy:
        src: "a_big_file"
        dest: "/python_scripts/backup"

EXPECTED RESULTS

On error, the copy module will remove the source files.

ACTUAL RESULTS

On remote target:

# find /root/.ansible/tmp/ -exec ls -ldh {} \;
drwx------. 2 root root 6 Nov  4 13:58 /root/.ansible/tmp/

On controller node:

# ansible-playbook -i inventory copy_test.yml -vvvv
ansible-playbook 2.8.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/Users/jowestco/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jowestco/Library/Python/venvs/ansible-latest/lib/python3.7/site-packages/ansible
  executable location = /Users/jowestco/Library/Python/venvs/ansible-latest/bin/ansible-playbook
  python version = 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) [Clang 6.0 (clang-600.0.57)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /Users/jowestco/clients/duke-energy/copy_test/inventory as it did not pass it's verify_file() method
script declined parsing /Users/jowestco/clients/duke-energy/copy_test/inventory as it did not pass it's verify_file() method
auto declined parsing /Users/jowestco/clients/duke-energy/copy_test/inventory as it did not pass it's verify_file() method
Parsed /Users/jowestco/clients/duke-energy/copy_test/inventory inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /Users/jowestco/Library/Python/venvs/ansible-latest/lib/python3.7/site-packages/ansible/plugins/callback/default.py

PLAYBOOK: copy_test.yml *****************************************************************************************************************************************************************************************************************************************
Positional arguments: copy_test.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/Users/jowestco/clients/duke-energy/copy_test/inventory',)
forks: 5
1 plays in copy_test.yml

PLAY [Create a copy failure] ************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [copy] *****************************************************************************************************************************************************************************************************************************************************
task path: /Users/jowestco/clients/duke-energy/copy_test/copy_test.yml:6
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<10.0.1.87> (0, b'/root\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/Users/jowestco/.ansible/cp/0d8c2dde66" does not exist\r\ndebug2: ssh_connect_direct\r\ndebug1: Connecting to 10.0.1.87 [10.0.1.87] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9993 ms remain after connect\r\ndebug1: identity file /Users/jowestco/.ssh/id_rsa type 0\r\ndebug1: identity file /Users/jowestco/.ssh/id_rsa-cert type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_dsa type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_dsa-cert type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_ecdsa type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_ecdsa-cert type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_ed25519 type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_ed25519-cert type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_xmss type -1\r\ndebug1: identity file /Users/jowestco/.ssh/id_xmss-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_7.9\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_7.8\r\ndebug1: match: OpenSSH_7.8 pat OpenSSH* compat 0x04000000\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: Authenticating to 10.0.1.87:22 as \'root\'\r\ndebug3: hostkeys_foreach: reading file "/Users/jowestco/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type ECDSA in file /Users/jowestco/.ssh/known_hosts:137\r\ndebug3: load_hostkeys: loaded 1 keys from 10.0.1.87\r\ndebug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521\r\ndebug3: send packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug3: receive packet: type 20\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: local client KEXINIT proposal\r\ndebug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c\r\ndebug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa\r\ndebug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r\ndebug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com\r\ndebug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r\ndebug2: compression ctos: zlib@openssh.com,zlib,none\r\ndebug2: compression stoc: zlib@openssh.com,zlib,none\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug2: peer server KEXINIT proposal\r\ndebug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1\r\ndebug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519\r\ndebug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc\r\ndebug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc\r\ndebug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512\r\ndebug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512\r\ndebug2: compression ctos: none,zlib@openssh.com\r\ndebug2: compression stoc: none,zlib@openssh.com\r\ndebug2: languages ctos: \r\ndebug2: languages stoc: \r\ndebug2: first_kex_follows 0 \r\ndebug2: reserved 0 \r\ndebug1: kex: algorithm: curve25519-sha256@libssh.org\r\ndebug1: kex: host key algorithm: ecdsa-sha2-nistp256\r\ndebug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com\r\ndebug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com\r\ndebug3: send packet: type 30\r\ndebug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r\ndebug3: receive packet: type 31\r\ndebug1: Server host key: ecdsa-sha2-nistp256 SHA256:uWkCk1DyYjcBth1uvXVkAH9gkImuj9yQJaExpmH318A\r\ndebug3: hostkeys_foreach: reading file "/Users/jowestco/.ssh/known_hosts"\r\ndebug3: record_hostkey: found key type ECDSA in file /Users/jowestco/.ssh/known_hosts:137\r\ndebug3: load_hostkeys: loaded 1 keys from 10.0.1.87\r\ndebug1: Host \'10.0.1.87\' is known and matches the ECDSA host key.\r\ndebug1: Found key in /Users/jowestco/.ssh/known_hosts:137\r\ndebug3: send packet: type 21\r\ndebug2: set_newkeys: mode 1\r\ndebug1: rekey after 134217728 blocks\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug3: receive packet: type 21\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug2: set_newkeys: mode 0\r\ndebug1: rekey after 134217728 blocks\r\ndebug1: Will attempt key: /Users/jowestco/.ssh/id_rsa RSA SHA256:tlXRuCJ2b3//1z0ZVgi5CXP1VQwlrEhzfgzLD6QnvD4\r\ndebug1: Will attempt key: /Users/jowestco/.ssh/id_dsa \r\ndebug1: Will attempt key: /Users/jowestco/.ssh/id_ecdsa \r\ndebug1: Will attempt key: /Users/jowestco/.ssh/id_ed25519 \r\ndebug1: Will attempt key: /Users/jowestco/.ssh/id_xmss \r\ndebug2: pubkey_prepare: done\r\ndebug3: send packet: type 5\r\ndebug3: receive packet: type 7\r\ndebug1: SSH2_MSG_EXT_INFO received\r\ndebug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,null>\r\ndebug3: receive packet: type 6\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 51\r\ndebug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r\ndebug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup gssapi-with-mic\r\ndebug3: remaining preferred: gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup gssapi-keyex\r\ndebug3: remaining preferred: hostbased,publickey\r\ndebug2: Unrecognized authentication method name: gssapi-keyex\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering public key: /Users/jowestco/.ssh/id_rsa RSA SHA256:tlXRuCJ2b3//1z0ZVgi5CXP1VQwlrEhzfgzLD6QnvD4\r\ndebug3: send packet: type 50\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug3: receive packet: type 60\r\ndebug1: Server accepts key: /Users/jowestco/.ssh/id_rsa RSA SHA256:tlXRuCJ2b3//1z0ZVgi5CXP1VQwlrEhzfgzLD6QnvD4\r\ndebug3: sign_and_send_pubkey: RSA SHA256:tlXRuCJ2b3//1z0ZVgi5CXP1VQwlrEhzfgzLD6QnvD4\r\ndebug3: sign_and_send_pubkey: signing using rsa-sha2-512\r\ndebug3: send packet: type 50\r\ndebug3: receive packet: type 52\r\ndebug1: Enabling compression at level 6.\r\ndebug1: Authentication succeeded (publickey).\r\nAuthenticated to 10.0.1.87 ([10.0.1.87]:22).\r\ndebug1: setting up multiplex master socket\r\ndebug3: muxserver_listen: temporary control path /Users/jowestco/.ansible/cp/0d8c2dde66.Unsdh2ADPLseMHrt\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug3: fd 4 is O_NONBLOCK\r\ndebug3: fd 4 is O_NONBLOCK\r\ndebug1: channel 0: new [/Users/jowestco/.ansible/cp/0d8c2dde66]\r\ndebug3: muxserver_listen: mux listener channel 0 fd 4\r\ndebug2: fd 3 setting TCP_NODELAY\r\ndebug3: ssh_packet_set_tos: set IP_TOS 0x20\r\ndebug1: control_persist_detach: backgrounding master process\r\ndebug2: control_persist_detach: background process is 8204\r\ndebug2: fd 4 setting O_NONBLOCK\r\ndebug1: forking to background\r\ndebug1: Entering interactive session.\r\ndebug1: pledge: id\r\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\r\ndebug1: multiplexing control connection\r\ndebug3: fd 5 is O_NONBLOCK\r\ndebug3: fd 5 is O_NONBLOCK\r\ndebug1: channel 1: new [mux-control]\r\ndebug3: channel_post_mux_listener: new mux channel 1 fd 5\r\ndebug3: mux_master_read_cb: channel 1: hello sent\r\ndebug2: set_control_persist_exit_time: cancel scheduled exit\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4\r\ndebug2: mux_master_process_hello: channel 1 slave version 4\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4\r\ndebug2: mux_master_process_alive_check: channel 1: alive check\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 104\r\ndebug2: mux_master_process_new_session: channel 1: request tty 0, X 0, agent 0, subsys 0, term "xterm-256color", cmd "/bin/sh -c \'echo ~root && sleep 0\'", env 1\r\ndebug3: mux_master_process_new_session: got fds stdin 6, stdout 7, stderr 8\r\ndebug2: fd 7 setting O_NONBLOCK\r\ndebug2: fd 8 setting O_NONBLOCK\r\ndebug1: channel 2: new [client-session]\r\ndebug2: mux_master_process_new_session: channel_new: 2 linked to control channel 1\r\ndebug2: channel 2: send open\r\ndebug3: send packet: type 90\r\ndebug3: receive packet: type 80\r\ndebug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0\r\ndebug3: receive packet: type 4\r\ndebug1: Remote: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\r\ndebug3: receive packet: type 4\r\ndebug1: Remote: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding\r\ndebug3: receive packet: type 91\r\ndebug2: channel_input_open_confirmation: channel 2: callback start\r\ndebug2: client_session2_setup: id 2\r\ndebug1: Sending environment.\r\ndebug1: Sending env LANG = en_US.UTF-8\r\ndebug2: channel 2: request env confirm 0\r\ndebug3: send packet: type 98\r\ndebug1: Sending command: /bin/sh -c \'echo ~root && sleep 0\'\r\ndebug2: channel 2: request exec confirm 1\r\ndebug3: send packet: type 98\r\ndebug3: mux_session_confirm: sending success reply\r\ndebug2: channel_input_open_confirmation: channel 2: callback done\r\ndebug2: channel 2: open confirm rwindow 0 rmax 32768\r\ndebug2: channel 2: rcvd adjust 2097152\r\ndebug3: receive packet: type 99\r\ndebug2: channel_input_status_confirm: type 99 id 2\r\ndebug2: exec request accepted on channel 2\r\ndebug3: receive packet: type 98\r\ndebug1: client_input_channel_req: channel 2 rtype exit-status reply 0\r\ndebug3: mux_exit_message: channel 2: exit message, exitval 0\r\ndebug3: receive packet: type 98\r\ndebug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0\r\ndebug2: channel 2: rcvd eow\r\ndebug2: channel 2: chan_shutdown_read (i0 o0 sock -1 wfd 6 efd 8 [write])\r\ndebug2: channel 2: input open -> closed\r\ndebug3: receive packet: type 96\r\ndebug2: channel 2: rcvd eof\r\ndebug2: channel 2: output open -> drain\r\ndebug2: channel 2: obuf empty\r\ndebug2: channel 2: chan_shutdown_write (i3 o1 sock -1 wfd 7 efd 8 [write])\r\ndebug2: channel 2: output drain -> closed\r\ndebug3: receive packet: type 97\r\ndebug2: channel 2: rcvd close\r\ndebug3: channel 2: will not send data after close\r\ndebug2: channel 2: send close\r\ndebug3: send packet: type 97\r\ndebug2: channel 2: is dead\r\ndebug2: channel 2: gc: notify user\r\ndebug3: mux_master_session_cleanup_cb: entering for channel 2\r\ndebug2: channel 1: rcvd close\r\ndebug2: channel 1: output open -> drain\r\ndebug2: channel 1: chan_shutdown_read (i0 o1 sock 5 wfd 5 efd -1 [closed])\r\ndebug2: channel 1: input open -> closed\r\ndebug2: channel 2: gc: user detached\r\ndebug2: channel 2: is dead\r\ndebug2: channel 2: garbage collecting\r\ndebug1: channel 2: free: client-session, nchannels 3\r\ndebug3: channel 2: status: The following connections are open:\r\n  #1 mux-control (t16 nr0 i3/0 o1/16 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\r\n  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1)\r\n\r\ndebug2: channel 1: obuf empty\r\ndebug2: channel 1: chan_shutdown_write (i3 o1 sock 5 wfd 5 efd -1 [closed])\r\ndebug2: channel 1: output drain -> closed\r\ndebug2: channel 1: is dead (local)\r\ndebug2: channel 1: gc: notify user\r\ndebug3: mux_master_control_cleanup_cb: entering for channel 1\r\ndebug2: channel 1: gc: user detached\r\ndebug2: channel 1: is dead (local)\r\ndebug2: channel 1: garbage collecting\r\ndebug1: channel 1: free: mux-control, nchannels 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug3: channel 1: status: The following connections are open:\r\n  #1 mux-control (t16 nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1)\r\n\r\ndebug2: Received exit status from master 0\r\ndebug2: set_control_persist_exit_time: schedule exit in 60 seconds\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676 `" && echo ansible-tmp-1572894157.144567-247285791952676="` echo /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676 `" ) && sleep 0'"'"''
<10.0.1.87> (0, b'ansible-tmp-1572894157.144567-247285791952676=/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> Attempting python interpreter discovery
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<10.0.1.87> (0, b'PLATFORM\nLinux\nFOUND\n/usr/bin/python\n/usr/bin/python3.6\n/usr/bin/python2.7\n/usr/libexec/platform-python\n/usr/bin/python3\n/usr/bin/python\nENDFOUND\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<10.0.1.87> (0, b'{"platform_dist_result": ["redhat", "8.0", "Ootpa"], "osrelease_content": "NAME=\\"Red Hat Enterprise Linux\\"\\nVERSION=\\"8.0 (Ootpa)\\"\\nID=\\"rhel\\"\\nID_LIKE=\\"fedora\\"\\nVERSION_ID=\\"8.0\\"\\nPLATFORM_ID=\\"platform:el8\\"\\nPRETTY_NAME=\\"Red Hat Enterprise Linux 8.0 (Ootpa)\\"\\nANSI_COLOR=\\"0;31\\"\\nCPE_NAME=\\"cpe:/o:redhat:enterprise_linux:8.0:GA\\"\\nHOME_URL=\\"https://www.redhat.com/\\"\\nBUG_REPORT_URL=\\"https://bugzilla.redhat.com/\\"\\n\\nREDHAT_BUGZILLA_PRODUCT=\\"Red Hat Enterprise Linux 8\\"\\nREDHAT_BUGZILLA_PRODUCT_VERSION=8.0\\nREDHAT_SUPPORT_PRODUCT=\\"Red Hat Enterprise Linux\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\"8.0\\"\\n"}\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /Users/jowestco/Library/Python/venvs/ansible-latest/lib/python3.7/site-packages/ansible/modules/files/stat.py
<10.0.1.87> PUT /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmp07q5y0id TO /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py
<10.0.1.87> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 '[10.0.1.87]'
<10.0.1.87> (0, b'sftp> put /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmp07q5y0id /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /root size 0\r\ndebug3: Looking up /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmp07q5y0id\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:8056\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 8056 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/ /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py && sleep 0'"'"''
<10.0.1.87> (0, b'', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 -tt 10.0.1.87 '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py && sleep 0'"'"''
<10.0.1.87> (0, b'\r\n{"changed": false, "stat": {"exists": true, "path": "/python_scripts/backup", "mode": "0755", "isdir": true, "ischr": false, "isblk": false, "isreg": false, "isfifo": false, "islnk": false, "issock": false, "uid": 0, "gid": 0, "size": 20, "inode": 132, "dev": 2065, "nlink": 3, "atime": 1572892559.7899613, "mtime": 1572893826.524876, "ctime": 1572893826.524876, "wusr": true, "rusr": true, "xusr": true, "wgrp": false, "rgrp": true, "xgrp": true, "woth": false, "roth": true, "xoth": true, "isuid": false, "isgid": false, "blocks": 0, "block_size": 4096, "device_type": 0, "readable": true, "writeable": true, "executable": true, "pw_name": "root", "gr_name": "root", "mimetype": "inode/directory", "charset": "binary", "version": "266691817", "attributes": [], "attr_flags": ""}, "invocation": {"module_args": {"path": "/python_scripts/backup", "follow": false, "get_checksum": true, "checksum_algorithm": "sha1", "get_mime": true, "get_attributes": true, "get_md5": null}}}\r\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 10.0.1.87 closed.\r\n')
Using module file /Users/jowestco/Library/Python/venvs/ansible-latest/lib/python3.7/site-packages/ansible/modules/files/stat.py
<10.0.1.87> PUT /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmp1wuy44aj TO /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py
<10.0.1.87> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 '[10.0.1.87]'
<10.0.1.87> (0, b'sftp> put /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmp1wuy44aj /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /root size 0\r\ndebug3: Looking up /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmp1wuy44aj\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:105 I:2\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:8067\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 8067 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/ /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py && sleep 0'"'"''
<10.0.1.87> (0, b'', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 -tt 10.0.1.87 '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py && sleep 0'"'"''
<10.0.1.87> (0, b'\r\n{"changed": false, "stat": {"exists": false}, "invocation": {"module_args": {"path": "/python_scripts/backup/a_big_file", "follow": false, "get_checksum": true, "checksum_algorithm": "sha1", "get_mime": true, "get_attributes": true, "get_md5": null}}}\r\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 10.0.1.87 closed.\r\n')
<10.0.1.87> PUT /Users/jowestco/clients/duke-energy/copy_test/a_big_file TO /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source
<10.0.1.87> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 '[10.0.1.87]'
<10.0.1.87> (0, b'sftp> put /Users/jowestco/clients/duke-energy/copy_test/a_big_file /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /root size 0\r\ndebug3: Looking up /Users/jowestco/clients/duke-energy/copy_test/a_big_file\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:8 O:131072 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:9 O:163840 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:10 O:196608 S:8192\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 32768 bytes at 98304\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 8 32768 bytes at 131072\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 9 32768 bytes at 163840\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 10 8192 bytes at 196608\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/ /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source && sleep 0'"'"''
<10.0.1.87> (0, b'', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /Users/jowestco/Library/Python/venvs/ansible-latest/lib/python3.7/site-packages/ansible/modules/files/copy.py
<10.0.1.87> PUT /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmpgbdpx3x4 TO /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_copy.py
<10.0.1.87> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 '[10.0.1.87]'
<10.0.1.87> (0, b'sftp> put /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmpgbdpx3x4 /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_copy.py\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 3 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /root size 0\r\ndebug3: Looking up /Users/jowestco/.ansible/tmp/ansible-local-8198lknsr_1m/tmpgbdpx3x4\r\ndebug3: Sent message fd 3 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_copy.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:12162\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 12162 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 10.0.1.87 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/ /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_copy.py && sleep 0'"'"''
<10.0.1.87> (0, b'', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.0.1.87> ESTABLISH SSH CONNECTION FOR USER: root
<10.0.1.87> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/Users/jowestco/.ansible/cp/0d8c2dde66 -tt 10.0.1.87 '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_copy.py && sleep 0'"'"''
<10.0.1.87> (1, b'\r\n{"msg": "Failed to replace file: b\'/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source\' to /python_scripts/backup/a_big_file: [Errno 28] No space left on device", "exception": "Traceback (most recent call last):\\n  File \\"/tmp/ansible_copy_payload_tkb20gsr/ansible_copy_payload.zip/ansible/module_utils/basic.py\\", line 2226, in atomic_move\\n    os.rename(b_src, b_dest)\\nOSError: [Errno 18] Invalid cross-device link: b\'/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source\' -> b\'/python_scripts/backup/a_big_file\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 550, in move\\n    os.rename(src, real_dst)\\nOSError: [Errno 18] Invalid cross-device link: b\'/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source\' -> b\'/python_scripts/backup/.ansible_tmpyy2q2q4ca_big_file\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n  File \\"/tmp/ansible_copy_payload_tkb20gsr/ansible_copy_payload.zip/ansible/module_utils/basic.py\\", line 2270, in atomic_move\\n    shutil.move(b_src, b_tmp_dest_name)\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 564, in move\\n    copy_function(src, real_dst)\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 263, in copy2\\n    copyfile(src, dst, follow_symlinks=follow_symlinks)\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 122, in copyfile\\n    copyfileobj(fsrc, fdst)\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 82, in copyfileobj\\n    fdst.write(buf)\\nOSError: [Errno 28] No space left on device\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n  File \\"/tmp/ansible_copy_payload_tkb20gsr/ansible_copy_payload.zip/ansible/module_utils/basic.py\\", line 2274, in atomic_move\\n    shutil.copy2(b_src, b_tmp_dest_name)\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 263, in copy2\\n    copyfile(src, dst, follow_symlinks=follow_symlinks)\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 122, in copyfile\\n    copyfileobj(fsrc, fdst)\\n  File \\"/usr/lib64/python3.6/shutil.py\\", line 82, in copyfileobj\\n    fdst.write(buf)\\nOSError: [Errno 28] No space left on device\\n", "failed": true, "invocation": {"module_args": {"src": "/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source", "dest": "/python_scripts/backup", "_original_basename": "a_big_file", "follow": false, "checksum": "e125b3b7525bd2a65d9bc7591f6cfedaf57a034a", "backup": false, "force": true, "content": null, "validate": null, "directory_mode": null, "remote_src": null, "local_follow": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "unsafe_writes": null}}}\r\n', b'OpenSSH_7.9p1, LibreSSL 2.7.3\r\ndebug1: Reading configuration data /Users/jowestco/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 48: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.0.1.87 is address\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8206\r\ndebug3: mux_client_request_session: session request sent\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to 10.0.1.87 closed.\r\n')
<10.0.1.87> Failed to connect to the host via ssh: OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/jowestco/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: resolve_canonicalize: hostname 10.0.1.87 is address
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 8206
debug3: mux_client_request_session: session request sent
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
Shared connection to 10.0.1.87 closed.
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_copy_payload_tkb20gsr/ansible_copy_payload.zip/ansible/module_utils/basic.py", line 2226, in atomic_move
    os.rename(b_src, b_dest)
OSError: [Errno 18] Invalid cross-device link: b'/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source' -> b'/python_scripts/backup/a_big_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/shutil.py", line 550, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: b'/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source' -> b'/python_scripts/backup/.ansible_tmpyy2q2q4ca_big_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/ansible_copy_payload_tkb20gsr/ansible_copy_payload.zip/ansible/module_utils/basic.py", line 2270, in atomic_move
    shutil.move(b_src, b_tmp_dest_name)
  File "/usr/lib64/python3.6/shutil.py", line 564, in move
    copy_function(src, real_dst)
  File "/usr/lib64/python3.6/shutil.py", line 263, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 122, in copyfile
    copyfileobj(fsrc, fdst)
  File "/usr/lib64/python3.6/shutil.py", line 82, in copyfileobj
    fdst.write(buf)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/ansible_copy_payload_tkb20gsr/ansible_copy_payload.zip/ansible/module_utils/basic.py", line 2274, in atomic_move
    shutil.copy2(b_src, b_tmp_dest_name)
  File "/usr/lib64/python3.6/shutil.py", line 263, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 122, in copyfile
    copyfileobj(fsrc, fdst)
  File "/usr/lib64/python3.6/shutil.py", line 82, in copyfileobj
    fdst.write(buf)
OSError: [Errno 28] No space left on device

[DEPRECATION WARNING]: Distribution redhat 8.0 on host 10.0.1.87 should use /usr/libexec/platform-python, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered 
platform python for this host. See https://docs.ansible.com/ansible/2.8/reference_appendices/interpreter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
fatal: [10.0.1.87]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "checksum": "e125b3b7525bd2a65d9bc7591f6cfedaf57a034a",
    "diff": [],
    "invocation": {
        "module_args": {
            "_original_basename": "a_big_file",
            "attributes": null,
            "backup": false,
            "checksum": "e125b3b7525bd2a65d9bc7591f6cfedaf57a034a",
            "content": null,
            "delimiter": null,
            "dest": "/python_scripts/backup",
            "directory_mode": null,
            "follow": false,
            "force": true,
            "group": null,
            "local_follow": null,
            "mode": null,
            "owner": null,
            "regexp": null,
            "remote_src": null,
            "selevel": null,
            "serole": null,
            "setype": null,
            "seuser": null,
            "src": "/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source",
            "unsafe_writes": null,
            "validate": null
        }
    },
    "msg": "Failed to replace file: b'/root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source' to /python_scripts/backup/a_big_file: [Errno 28] No space left on device"
}

PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************
10.0.1.87                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

On remote node:

# find /root/.ansible/tmp/ -exec ls -ldh {} \;
drwx------. 3 root root 59 Nov  4 13:59 /root/.ansible/tmp/
drwx------ 2 root root 70 Nov  4 13:59 /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676
-rwx------ 1 root root 104K Nov  4 13:59 /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_stat.py
*-rwx------ 1 root root 200K Nov  4 13:59 /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/source*
-rwx------ 1 root root 108K Nov  4 13:59 /root/.ansible/tmp/ansible-tmp-1572894157.144567-247285791952676/AnsiballZ_copy.py

The tmp directory is left lying around with the 200k file in it.

ansible / ansible