run_local not working on windows

[jacobEAdamson]

My system is Python 3.7.0 running Windows 10 and I receive an error when I try to run anything with run_local on windows (which affects all backends). The stack trace looks like this:

venv\lib\site-packages\testinfra\host.py:71: in run
    return self.backend.run(command, *args, **kwargs)
venv\lib\site-packages\testinfra\backend\docker.py:35: in run
    "docker exec %s /bin/sh -c %s", self.name, cmd)
venv\lib\site-packages\testinfra\backend\base.py:203: in run_local
    stderr=subprocess.PIPE,
..\..\..\appdata\local\programs\python\python37-32\Lib\subprocess.py:756: in __init__
    restore_signals, start_new_session)
..\..\..\appdata\local\programs\python\python37-32\Lib\subprocess.py:1100: in _execute_child
    args = list2cmdline(args)
..\..\..\appdata\local\programs\python\python37-32\Lib\subprocess.py:511: TypeError
    TypeError: argument of type 'int' is not iterable

Looks like the fact that the command is being encoded before running is to blame. Popen in Windows doesn't handle byte arrays the same way it does on Linux. Was able to fix by modifying code to this:

def run_local(self, command, *args):
    command = self.quote(command, *args)
    if os.name != 'nt':
        command = self.encode(command)
    p = subprocess.Popen(
        command, shell=True,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    stdout, stderr = p.communicate()
    result = self.result(p.returncode, command, stdout, stderr)
    return result

Basically only encode when not on windows

[ianw]

Thanks, from @jacobEAdamson comments in the pull request and seeing here you mention python 3.7 things are a little more clear, and a little more confused.

self.encode() is going to return a bytes object on python3 which will mean the following

$ python
Python 2.7.16 (default, Apr 30 2019, 15:54:43) 
>>> subprocess.list2cmdline(b"testing 123")
't e s t i n g " " 1 2 3'

$ python3
Python 3.7.4 (default, Jul  9 2019, 16:32:37) 
>>> subprocess.list2cmdline(b'testing 123')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.7/subprocess.py", line 530, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'int' is not iterable

So that explains the error above, but this raises more questions for me ...

It looks like this happened originally with 4ab331a82972d5dc7e77c7078fb7847e638bf5be which gives the clue it was related to something like ISO-8859-15 systems and a command like

Command("ls -l %s", "/é")

I think maybe the problem was that you're calling Popen("ssh remotehost ls -l /é") and so Popen encodes the "/é" for the local system (https://github.com/python/cpython/blob/v3.7.3/Lib/subprocess.py#L1436) but then that is incorrect when the remote is running something like ISO-8859-15, and it now gets a (say) UTF-8 encoded command-line to try and run? So by making it a bytes-like object you short-circuit python trying to encode for you?

Then there's a bunch of suff about CreateProcessA and CreateProcessW on windows that suggests that running commands should either be an ascii string or UTF-16 chars. Then I just get lost ... :)

[KiraUnderwood]

Same problem. Only python 2.7 seems to work. I've tried 3.5 and 3.7 as well and got the above mentioned: needquote = (" " in arg) or ("\t" in arg) or not arg TypeError: argument of type 'int' is not iterable_

[viniciusartur]

Similar problem. Windows 2008 r2 + Python 3.8.3 + testinfra 5.2.1. I got the error bytes args is not allowed on Windows when trying to use run_local:

Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 b
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import testinfra
>>> host = testinfra.host.get_host('local://')
>>> host.run("echo")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\
ackages\testinfra\host.py", line 75, in run
    return self.backend.run(command, *args, **kwargs)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\
ackages\testinfra\backend\local.py", line 30, in run
    return self.run_local(self.get_command(command, *args))
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\
ackages\testinfra\backend\base.py", line 192, in run_local
    p = subprocess.Popen(
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\
cess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\
cess.py", line 1239, in _execute_child
    raise TypeError('bytes args is not allowed on Windows')
TypeError: bytes args is not allowed on Windows

The workaround for me is a runtime method patching removing the self.encode(command).

[turboscholz]

Hey Vinicius,

The workaround for me is a runtime method patching removing the self.encode(command).

Can you show me how you did this? I've tried a lot of things now, but I did not succeed. This is my test_infra.py file:

import sys

sys.path.append('C:\\Python39\\Lib\\site-packages\\testinfra\\backend\\')

from base import BaseBackend

def run_local_new(self, command, *args):
   command = self.quote(command, *args)
   p = subprocess.Popen(
       command, shell=True,
       stdin=subprocess.PIPE,
       stdout=subprocess.PIPE,
       stderr=subprocess.PIPE,
   )
   stdout, stderr = p.communicate()
   result = self.result(p.returncode, command, stdout, stderr)
   return result

BaseBackend.run_local = run_local_new

def test_puppet_facts(host):
    facts = host.facter()
    assert facts["operatingsystem"] == "windows"

But when running the test via

C:\Python39\python -m pytest -v .\test_infra.py

I still get

C:\Python39\lib\site-packages\testinfra\modules\puppet.py:105: in __call__
    return json.loads(self.check_output(cmd))
c:\python39\lib\site-packages\testinfra\host.py:75: in run
    return self.backend.run(command, *args, **kwargs)
c:\python39\lib\site-packages\testinfra\backend\local.py:30: in run
    return self.run_local(self.get_command(command, *args))
C:\Python39\lib\site-packages\testinfra\backend\base.py:192: in run_local
    p = subprocess.Popen(
C:\Python39\lib\subprocess.py:947: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
[...]
TypeError: bytes args is not allowed on Windows

How did you patch the run_local(self, command, *args) method and how can I do this with the approach above? For me it looks like as if my patched run_local method is not used.

[viniciusartur]

How did you patch the run_local(self, command, *args) method and how can I do this with the approach above? For me it looks like as if my patched run_local method is not used.

In my conftest.py:

def patch_testinfra():
    def run_local_new(self, command, *args):
        command = self.quote(command, *args)
        p = subprocess.Popen(
            command, shell=True,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        stdout, stderr = p.communicate()
        result = self.result(p.returncode, command, stdout, stderr)
        return result

    from testinfra.backend.base import BaseBackend
    BaseBackend.run_local = run_local_new

def pytest_generate_tests(metafunc):
    patch_testinfra()

[thomasleveil]

@viniciusartur solution wasn't suffisent in my case (I'm using testinfra with docker backend), but combining its solution with https://github.com/pytest-dev/pytest-testinfra/issues/375#issue-360991022 made it work :

# conftest.py
import os
import re
import subprocess
import testinfra

def patch_testinfra():
    if os.name == 'nt':
        """
        Making testinfa work from Windows with Docker backend.
        Inspired from 
        - https://github.com/pytest-dev/pytest-testinfra/issues/411#issuecomment-782729362
        - https://github.com/pytest-dev/pytest-testinfra/issues/375#issue-360991022 
        """
        def quote(command, *args):
            def anon_1(arg):
                if re.match(r'/("|\s|\')', arg) != None:
                    return arg

                arg = re.sub('"', '\"', arg)
                return '"%s"' % (arg)

            if args:
                return command % tuple(anon_1(a) for a in args)
                # return command % tuple(pipes.quote(a) for a in args)
            return command

        def run_local_new(self, command, *args):
            command = quote(command, *args)
            p = subprocess.Popen(
                command,
                shell=True,
                stdin=subprocess.PIPE,
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
            )
            stdout, stderr = p.communicate()
            result = self.result(p.returncode, command, stdout, stderr)
            return result

        from testinfra.backend.base import BaseBackend

        BaseBackend.run_local = run_local_new

def pytest_generate_tests(metafunc):
    patch_testinfra()