lmstudio-ai / venvstacks

Virtual environment stacks for Python
http://venvstacks.lmstudio.ai/
MIT License
172 stars 5 forks source link

Windows: direct execution test fails in CI #42

Closed ncoghlan closed 3 weeks ago

ncoghlan commented 1 month ago

In #41 test_entry_point_help fails in the Windows CI with:

  __________________ TestTopLevelCommand.test_entry_point_help __________________

  self = <test_cli_invocation.TestTopLevelCommand object at 0x000001FBC6F519D0>

      def test_entry_point_help(self) -> None:
          if sys.prefix == sys.base_prefix:
              pytest.skip("Entry point test requires test execution in venv")
          expected_entry_point = Path(sys.executable).parent / "venvstacks"
          command = [str(expected_entry_point), "--help"]
          result = run_python_command_unchecked(
              command, stdout=subprocess.PIPE, stderr=subprocess.PIPE
          )
          # Usage message should suggest direct execution
  >       assert "Usage: venvstacks [" in result.stdout
  E       TypeError: argument of type 'NoneType' is not iterable

  tests\test_cli_invocation.py:160: TypeError
  ============================== warnings summary ===============================
  tests/test_cli_invocation.py::TestTopLevelCommand::test_entry_point_help
    D:\a\venvstacks\venvstacks\.tox\py3.12\Lib\site-packages\_pytest\threadexception.py:82: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-1 (_readerthread)

    Traceback (most recent call last):
      File "C:\hostedtoolcache\windows\Python\3.12.7\x64\Lib\threading.py", line 1075, in _bootstrap_inner
        self.run()
      File "C:\hostedtoolcache\windows\Python\3.12.7\x64\Lib\threading.py", line 1012, in run
        self._target(*self._args, **self._kwargs)
      File "C:\hostedtoolcache\windows\Python\3.12.7\x64\Lib\subprocess.py", line 1599, in _readerthread
        buffer.append(fh.read())
                      ^^^^^^^^^
      File "C:\hostedtoolcache\windows\Python\3.12.7\x64\Lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 623: character maps to <undefined>

This isn't a general Windows command execution problem, since venvstacks runs a lot of subcommands of its own, and the project tests run various queries in the target Python environments that will include Unicode characters in their results.

It's being worked around with an xfail for now, since this is a stock standard entry point definition, so the problem is almost certainly in the test suite or the test environment setup.

ncoghlan commented 1 month ago

And the whole reason this test case uses run_python_command_unchecked is so it gets the same "Keep subprocesses from screwing up their text encodings" as venvstacks itself uses: https://github.com/lmstudio-ai/venvstacks/blob/bb1c7e9b2eb9ba7f0df36a6950cfa92aaefdaf23/src/venvstacks/_util.py#L63

ncoghlan commented 3 weeks ago

Turned out this was a real problem with the encoding handling when communicating with subprocesses: https://github.com/lmstudio-ai/venvstacks/pull/46

The reason the Windows CI was the one that picked it up is because it defaults to cp1252 instead of utf-8.