search

Rahix / tbot

Automation/Testing tool for Embedded Linux Development
https://tbot.tools
GNU General Public License v3.0
89 stars 21 forks source link

Tbot Selftest fails in virtual environment #51

Closed NoUmlautsAllowed closed 3 years ago

NoUmlautsAllowed commented 3 years ago

The tbot selftest selftest_machine_labhost_shell fails if tbot is executed in a virtual environment. Running tbot outside the virtual environment is successful.

Steps to reproduce:

  1. Have tbot installed with pip3 install -U --user git+https://github.com/rahix/tbot@v0.9.0
  2. Create a virtual environment python -m venv venv
  3. Activate the virtual environment source venv/bin/activate
  4. Run tbot selftest

Tested with Python version Python 3.8.5 and Python 3.9.5

Sample output:

$ tbot selftest
tbot starting ...
├─Calling selftest ...
│   ├─Calling testsuite ...
│   │   ├─Calling selftest_failing ...
│   │   │   ├─Calling inner ...
│   │   │   │   └─Fail. (0.000s)
│   │   │   └─Done. (0.000s)
│   │   ├─Calling selftest_skipping ...
│   │   │   ├─Calling inner ...
│   │   │   │   └─Skipped: This test is skipped on purpose
│   │   │   └─Done. (0.000s)
│   │   ├─Calling selftest_uname ...
│   │   │   └─Done. (0.002s)
│   │   ├─Calling selftest_user ...
│   │   │   └─Done. (0.001s)
│   │   ├─Calling selftest_machine_reentrant ...
│   │   │   └─Done. (0.000s)
│   │   ├─Calling selftest_machine_labhost_shell ...
│   │   │   ├─Calling selftest_machine_shell ...
│   │   │   │   ├─Testing command output ...
│   │   │   │   ├─Testing return codes ...
│   │   │   │   ├─Testing env vars ...
│   │   │   │   ├─Testing redirection (and weird paths) ...
│   │   │   │   ├─99895
│   │   │   │   ├─Testing subshell ...
│   │   │   │   ├─Testing mach.run() ...
│   │   │   │   └─Fail. (0.034s)
│   │   │   └─Fail. (0.034s)
│   │   ├─Calling selftest_machine_ssh_shell ...
│   │   │   ├─Calling check_minisshd ...
│   │   │   │   └─Done. (0.002s)
│   │   │   └─Skipped: dropbear is not installed so ssh can't be tested
│   │   ├─Calling selftest_machine_sshlab_shell ...
│   │   │   ├─Calling check_minisshd ...
│   │   │   │   └─Done. (0.002s)
│   │   │   └─Skipped: dropbear is not installed so ssh can't be tested
│   │   ├─Calling selftest_path_stat ...
│   │   │   ├─Setting up test files ...
│   │   │   ├─Checking existence ...
│   │   │   ├─Checking file modes ...
│   │   │   ├─Checking file modes on nonexistent files ...
│   │   │   ├─Checking stat results ...
│   │   │   └─Done. (0.022s)
│   │   ├─Calling selftest_path_integrity ...
│   │   │   └─Done. (0.047s)
│   │   ├─Calling selftest_path_files ...
│   │   │   ├─Testing text file access ...
│   │   │   ├─Testing binary file access ...
│   │   │   ├─Test reading/writing invalid file ...
│   │   │   └─Done. (0.024s)
│   │   ├─Calling selftest_board_power ...
│   │   │   ├─Emulating a normal run ...
│   │   │   ├─POWERON (test-ub-power)
│   │   │   ├─UBOOT (test-ub-power)
│   │   │   ├─POWEROFF (test-ub-power)
│   │   │   ├─Emulating a failing run ...
│   │   │   ├─POWERON (test-ub-power)
│   │   │   ├─UBOOT (test-ub-power)
│   │   │   ├─raise TestException()
│   │   │   ├─POWEROFF (test-ub-power)
│   │   │   └─Done. (0.246s)
│   │   ├─Calling selftest_board_uboot ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─Calling selftest_machine_shell ...
│   │   │   │   ├─Testing command output ...
│   │   │   │   ├─Testing return codes ...
│   │   │   │   ├─Testing env vars ...
│   │   │   │   └─Done. (0.008s)
│   │   │   └─Done. (0.237s)
│   │   ├─Calling selftest_board_uboot_noab ...
│   │   │   ├─UBOOT (test-ub-noab)
│   │   │   ├─Calling selftest_machine_shell ...
│   │   │   │   ├─Testing command output ...
│   │   │   │   ├─Testing return codes ...
│   │   │   │   ├─Testing env vars ...
│   │   │   │   └─Done. (0.009s)
│   │   │   └─Done. (0.431s)
│   │   ├─Calling selftest_board_linux ...
│   │   │   └─Skipped: No board available
│   │   ├─Calling selftest_board_linux_uboot ...
│   │   │   ├─Testing without UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   ├─Testing with UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   └─Done. (0.866s)
│   │   ├─Calling selftest_board_linux_standalone ...
│   │   │   ├─Testing without UB ...
│   │   │   ├─LINUX (test-board-linux-standalone)
│   │   │   ├─Testing with UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   └─Done. (1.446s)
│   │   ├─Calling selftest_board_linux_nopw ...
│   │   │   ├─Testing without UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub_nopw)
│   │   │   ├─Testing with UB ...
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub_nopw)
│   │   │   └─Done. (0.463s)
│   │   ├─Calling selftest_board_linux_bad_console ...
│   │   │   └─Skipped: board-linux bad console test is not implemented
│   │   ├─Calling selftest_with_lab ...
│   │   │   ├─Calling selftest_decorated_lab ...
│   │   │   │   └─Done. (0.022s)
│   │   │   ├─Calling selftest_decorated_lab ...
│   │   │   │   └─Done. (0.001s)
│   │   │   └─Done. (0.024s)
│   │   ├─Calling selftest_with_uboot ...
│   │   │   ├─Calling selftest_decorated_uboot ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   └─Done. (0.440s)
│   │   │   ├─Calling selftest_decorated_uboot ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   └─Done. (0.620s)
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─Calling selftest_decorated_uboot ...
│   │   │   │   └─Done. (0.001s)
│   │   │   └─Done. (1.480s)
│   │   ├─Calling selftest_with_linux ...
│   │   │   ├─Calling selftest_decorated_linux ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   │   └─Done. (0.051s)
│   │   │   ├─Calling selftest_decorated_linux ...
│   │   │   │   ├─UBOOT (test-ub)
│   │   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   │   └─Done. (1.033s)
│   │   │   ├─UBOOT (test-ub)
│   │   │   ├─LINUX (test-board-linux-ub)
│   │   │   ├─Calling selftest_decorated_linux ...
│   │   │   │   └─Done. (0.002s)
│   │   │   └─Done. (1.114s)
│   │   ├─────────────────────────────────────────
│   │   │ Failure: 1/22 tests failed
│   │   │ 
│   │   ├─selftest_machine_labhost_shell:
│   │   │ Traceback (most recent call last):
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/__init__.py", line 43, in testsuite
│   │   │     test(**kwargs)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│   │   │     return tc(*args, **kwargs)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/selftest/machine.py", line 54, in selftest_machine_labhost_shell
│   │   │     selftest_machine_shell(lh)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│   │   │     return tc(*args, **kwargs)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/selftest/machine.py", line 258, in selftest_machine_shell
│   │   │     bs.terminate0()
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/machine/linux/util.py", line 160, in terminate0
│   │   │     retcode, output = self.terminate()
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/machine/linux/util.py", line 176, in terminate
│   │   │     next(self._cmd_context)
│   │   │   File "/home/nou/.local/lib/python3.9/site-packages/tbot/machine/linux/bash.py", line 206, in cmd_context
│   │   │     retcode = int(proxy_ch.read_until_prompt())
│   │   │ ValueError: invalid literal for int() with base 10: '2004l\rexit\necho $?\n'
│   │   │ 
│   │   └─Fail. (6.444s)
│   └─Fail. (6.665s)
├─Exception:
│   Traceback (most recent call last):
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/main.py", line 345, in main
│       func(**params)
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│       return tc(*args, **kwargs)
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/selftest/__init__.py", line 80, in selftest
│       tc.testsuite(
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/decorators.py", line 62, in wrapped
│       return tc(*args, **kwargs)
│     File "/home/nou/.local/lib/python3.9/site-packages/tbot/tc/__init__.py", line 62, in testsuite
│       raise Exception(f"{len(errors)}/{len(args)} tests failed")
│   Exception: 1/22 tests failed
├─────────────────────────────────────────
└─FAILURE (6.721s)
Rahix commented 3 years ago

Hi,

the error message gives a clue that there appear to be some unexpected escape sequences in the output:

invalid literal for int() with base 10: '2004l\rexit\necho $?\n'

The interesting part is the 2004l and that's an escape sequence related to bracketed paste mode. I really have no clue how this could be related to using a venv or not, but I did notice that I can produce a similar failure on my system when using the new pytest testsuite:

❯ python -m pytest selftest/ -k 'test_simple_output' -v
===================================== test session starts ======================================
platform linux -- Python 3.9.4, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/rahix/Documents/Development/tbot
plugins: hypothesis-6.10.1, mock-3.1.1, qt-3.3.0
collected 138 items / 134 deselected / 4 selected                                              

selftest/tests/test_shell.py::test_simple_output[LocalhostBash] PASSED                   [ 25%]
selftest/tests/test_shell.py::test_simple_output[LocalhostSlowBash] PASSED               [ 50%]
selftest/tests/test_shell.py::test_simple_output[LocalhostAsh] FAILED                    [ 75%]
selftest/tests/test_shell.py::test_simple_output[MocksshClient] PASSED                   [100%]

=========================================== FAILURES ===========================================
_______________________________ test_simple_output[LocalhostAsh] _______________________________

[...]

>       return (int(retcode), out)
E       ValueError: invalid literal for int() with base 10: '\x1b[?2004l\r0\n\x1b[?2004h'

It is clearly visible here that the other end disabled bracketed paste mode after entering a command and reenabled it before returing to the prompt. So this smells a lot like GNU readline and indeed the changelog for readline version 8.1 notes that

h. Bracketed paste mode is enabled by default. There is a configure-time option (--enable-bracketed-paste-default) to set the default to on or off.

Now, this leaves me wondering why it even works at all with bash right now, but apparently LocalhostBash works fine and just LocalhostAsh (which uses bash --posix under the hood) does not. I need to take a closer look to find out what's going on here... In the meantime, can you verify that you have GNU readline version 8.1 installed?

Rahix commented 3 years ago

Okay, I found a solution that at least solves the problems I could see. Please try again with the latest version from master (ec0f6c6df11e ("treewide: Invoke bash with --noediting")) whether the problem is fixed for you as well.

NoUmlautsAllowed commented 3 years ago

Hi,

thanks for the quick response.

Yes, I can confirm that I have GNU readline 8.1 installed. The changes in ec0f6c6df11e0722d701f32f18eed96e2aef9b66 fix the problem for me as well.

Thanks.

Adnan-Elhammoudi commented 10 months ago

it can be solved by using the pip inside the environment and then install the tbot

something like this :


python3 -m venv venv
 venv/bin/activate
python3 -m ensurepip --upgrade
python3 -m pip install -U git+https://github.com/rahix/tbot@v0.10.6