search

microsoft / vscode-python

Python extension for Visual Studio Code
https://aka.ms/pvsc-marketplace
MIT License
4.33k stars 1.19k forks source link

pytest parameterized tests with randomly generate names #22648

Closed cquick01 closed 10 months ago

cquick01 commented 10 months ago

Type: Bug

Behaviour

Expected vs. Actual

When running pytest via VSCode, I get the following errors

Running tests (pytest): /home/user/test/tests/test_bytes.py::test_randbytes
Running test with arguments: --rootdir /home/user/test --override-ini junit_family=xunit1 --junit-xml=/tmp/tmp-522739qc0UGxP1z6FD.xml --capture no ./tests/test_bytes.py::test_randbytes
Current working directory: /home/user/test
Workspace directory: /home/user/test
Run completed, parsing output
Test result not found for: ./tests/test_bytes.py::test_randbytes[b8\xef\xef]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xb4d\xed\xaek\xac\xdbR\xde\xe2\x87\xcc5\x9c]\x95\x02.\xac\xd1)\x8a\xb3\xaf\x15\x87\xad\x83c\x0b\x94Z\x8d\x03\xcd<\xd6\xa0\xe9\xba\x11\x95]

Test result not found for: ./tests/test_bytes.py::test_randbytes[5?5\x9cIGQG\xba\x8a\xdb\xf7\xc8\xec k\x95\x08\xbb \x8c\x1eBb\xd33\x93\x1c\x93\xe6\xc2\x08\x7f\xd1\xa7\x95\xf5m\xca\xaa\xe2\x90\x8c\x97\x80\xfe\xd0\xd2#\xbdI\x1dV3>\x8e7\x01\xffg\r]

Test result not found for: ./tests/test_bytes.py::test_randbytes[k[\x02M\xe2\r*\xcd4\xc2Q+\xadbqP\xca\xac\xc4]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xdb\x1d\xe8\x04\n\x9b\x0b/.\xd55\xd2\x92\xbf\x05/\xc1\xa0\x06\x95\xd2s\x18\xd1\xed'\xd9Cu\x02S\xe1\x0b\xc4\xa6\x00)\xda\xf1]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xe4\x94`\xd0\xb9V\x97g%\xafa\xda\xc8\xd1\xbd\xd7\xc5Z\xf7\xb7\xdf\xbdd]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xb2\x8f]

Test result not found for: ./tests/test_bytes.py::test_randbytes[z:\x1f\xccq`\xd6\x88\xef\x8a\xbc*\x8f\x95\x8e\x8c\xf9O\xf1\xdbo(]

Test result not found for: ./tests/test_bytes.py::test_randbytes[\xd5h[\x7f\xe2\xf6m\xf1]

Test result not found for: ./tests/test_bytes.py::test_randbytes[_\x82p\xac\x07\xc1\xdbO\n\xe1<\x19\xb2\xbbW\x18\x95\x02H\xc8]

Total number of tests expected to run: 10
Total number of tests run: 0
Total number of tests passed: 0
Total number of tests failed: 0
Total number of tests failed with errors: 0
Total number of tests skipped: 0
Total number of tests with no result data: 10
Finished running tests!

But the debug window shows that it passes

============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/user/test
collected 10 items

tests/test_bytes.py ..........

------------- generated xml file: /tmp/tmp-522739qc0UGxP1z6FD.xml --------------
============================== 10 passed in 0.10s ==============================

And it works fine when running on command line.

Steps to reproduce:

  1. Setup Environment
.
├── poetry.lock
├── pyproject.toml
└── tests
    └── test_bytes.py

test_bytes.py

#!/usr/bin/env python3
import random

import pytest

RANDOM_BYTES = [random.randbytes(random.randint(0, 64)) for _ in range(10)]

@pytest.mark.parametrize("data", RANDOM_BYTES)
def test_randbytes(data):
    foo = data
    assert foo == data

pyproject.toml

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["User <user@domain.tld>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
pytest = "^7.4.3"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
  1. Install with poetry
pip install poetry  # if needed
poetry lock
poetry install
  1. Configure Python interpreter in VSCode
# to get path of environment
poetry env info -p
  1. Run tests, errors occur. Running pytest in terminal passes.

Edit: Also, when adding this test with ints, most fail the same way, but two of them seem to pass.

image

But the debug window still shows all tests pass, 20 this time

============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/cquick/test
collected 20 items

tests/test_bytes.py ....................

------------- generated xml file: /tmp/tmp-522739BQWWvKOgqoHN.xml --------------
============================== 20 passed in 0.10s ==============================

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

``` 2023-12-13 17:19:53.199 [debug] Clearing context for python dependencies not installed: /home/user/test/pyproject.toml 2023-12-13 17:19:53.202 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 2023-12-13 17:19:53.202 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 2023-12-13 17:19:53.219 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 2023-12-13 17:19:53.234 [info] > ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 -m pytest --rootdir . --override-ini junit_family=xunit1 --junit-xml=/tmp/tmp-5227399oU3WLyyfScP.xml 2023-12-13 17:19:53.234 [info] cwd: . 2023-12-13 17:19:57.440 [debug] Clearing context for python dependencies not installed: ms-python.python.Python 2023-12-13 17:19:57.440 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 2023-12-13 17:20:03.697 [debug] Clearing context for python dependencies not installed: /home/user/test/pyproject.toml 2023-12-13 17:20:03.700 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 2023-12-13 17:20:11.268 [debug] Clearing context for python dependencies not installed: ms-python.python.Python 2023-12-13 17:20:11.268 [debug] Found cached env for /home/user/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 ```

User Settings

``` languageServer: "Pylance" testing • pytestEnabled: true ```

Extension version: 2023.23.13471011 VS Code version: Code - Insiders 1.86.0-insider (f87fb292c3af22e51a7f5c628496cd8d6fd9ecd5, 2023-12-13T08:07:18.340Z) OS version: Windows_NT x64 10.0.19045 Modes: Remote OS version: Linux x64 5.15.133.1-microsoft-standard-WSL2

System Info |Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2592)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled| |Load (avg)|undefined| |Memory (System)|31.70GB (5.56GB free)| |Process Argv|| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|WSL: Ubuntu-22.04| |OS|Linux x64 5.15.133.1-microsoft-standard-WSL2| |CPUs|Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2592)| |Memory (System)|11.68GB (6.36GB free)| |VM|0%|
eleanorjboyd commented 10 months ago

Hello, it looks like you are not on the new testing rewrite. Could you enable that and try this again?

It should be turned on for everyone but you might have disabled all experiments so could you check there? If not you could try adding this to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. You can check if its enabled by looking at Experiment 'pythonTestAdapter' is active in your python logs. If you are unable to see the log for the experiment change your log level to info or trace and it should show up (can be done via the command palette). Thanks!

cquick01 commented 10 months ago

Thanks for the response @eleanorjboyd . I enabled the pythonTestAdapter experiment but am still running into issues running the tests.

I see that the experiment is now activated

2023-12-15 09:48:03.524 [info] Telemetry level is off
2023-12-15 09:48:03.524 [info] Experiments are disabled, only manually opted experiments are active.
2023-12-15 09:48:03.524 [info] Experiment 'pythonTestAdapter' is active
2023-12-15 09:48:03.722 [info] Default formatter is set to null for workspace /home/user/test
2023-12-15 09:48:04.805 [info] Test server listening.
2023-12-15 09:48:05.027 [info] > ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3 -I ~/.vscode-server-insiders/extensions/ms-python.python-2023.23.13481009/pythonFiles/get_output_via_markers.py ~/.vscode-server-insiders/extensions/ms-python.python-2023.23.13481009/pythonFiles/interpreterInfo.py
2023-12-15 09:48:05.667 [info] Python interpreter path: ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/python3
2023-12-15 09:48:05.968 [info] > conda info --json
2023-12-15 09:48:06.339 [info] > . ~/.cache/pypoetry/virtualenvs/test-zxyBamcN-py3.11/bin/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode-server-insiders/extensions/ms-python.python-2023.23.13481009/pythonFiles/printEnvVariables.py
2023-12-15 09:48:06.339 [info] shell: bash
2023-12-15 09:48:07.403 [info] Starting Pylance language server.

The tests still show errors, although they seem to be different now

CLIENT: Server listening on port 41559...
Received JSON data in run script
Running pytest with args: ['-p', 'vscode_pytest', '--rootdir', '/home/user/test', '/home/user/test/tests/test_bytes.py::test_randints[17]', '/home/user/test/tests/test_bytes.py::test_randints[46]', '/home/user/test/tests/test_bytes.py::test_randints[6]', '/home/user/test/tests/test_bytes.py::test_randints[18]', '/home/user/test/tests/test_bytes.py::test_randints[0]', '/home/user/test/tests/test_bytes.py::test_randints[48]', '/home/user/test/tests/test_bytes.py::test_randints[14]', '/home/user/test/tests/test_bytes.py::test_randints[60]', '/home/user/test/tests/test_bytes.py::test_randints[39]', '/home/user/test/tests/test_bytes.py::test_randints[64]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x80\\xde}=\\x8c2/7\\x00Y\\x10`\\x0b\\xce%\\xfd\\xbd\\xd0_\\xee\\x11\\xe8rW\\x8c\\xb9\\xf0N\\xa11<f\\xc9u\\xb3\\x92\\x18\\x0c\\xeb\\x99\\x136?\\x13\\xc7\\xda\\xbe\\xee\\xc1t\\x87\\xeaE\\xa7\\xebk\\x14\\x86\\xcdv]', '/home/user/test/tests/test_bytes.py::test_randbytes[c\\xd9\\xd9\\x98|e`V\\xc3\\xce\\x04N\\xfc*\\xfc)o\\xb5\\xa6\\xe6\\xedJ\\x0eo\\x94\\xea\\x17\\xef]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x9b\\xe4\\xf63yd\\xe1)\\x02\\x87\\x17\\xd3\\xc4\\xc7M\\xacc\\xab\\x1cdJ\\xca\\x04\\\\x81\\n\\xca.\\xe7\\x9f\\x13\\xfd\\xe3\\x86\\xa2c\\xe2\\xfe\\xaf]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x05\\xd5\\xed\\x12\\xa4!e\\xaf\\x16$\\xfb"<\\x91\\xce\\xd9\\xb9\\xdb`\\xbe\\xfd:\\xf0\\xd8/,\\x84\\xbdC\\xf7\\xdd\\x0c\\x94?\\xe4\\xe0\\xbe]\\xa2\\xb53\\xf6\\xb80\\x93\\xfd\\xbabC\\xfc\\xba\\xe5s<\\x00d]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xb8\\x0e\\xfcs\\xa2]\\xfe\\xb2\\xbe\\x06\\xd4\\x83\\x8e\\x9a\\xc1\\xee\\xd7K\\nL=^\\xb6\\x99\\xce1N\\xf5\\x06\\x9f\\xebM\\xf0\\xfb\\x9c\\xc0\\x81\\x04i\\x10\\x7f\\xa9]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x92k\\x11\\x1c\\xb0m\\x89\\x16j\\x02"\\xf3\\xf2\\xc9\\xcb\\x9aG]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xec"\\nZ\\xbc\\xd5T\\xb1\\xeczX\\xd1\\xee]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xe2\\xa4\\x84&\\xb2[Hk\\xc67\\xaf\\x13\\x96\\x18\\xa6\\xbd\\x0bC\\x1cS\\xd1\\xde\\x9e\\xfeJ\\x87,\\x00L\\xefr\\xc3\\xfb-\\x81\\xdb\\xf9\\xc5]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\xdf\\xab)u\\x14\\xe2\\x8d\\xe0\\xd8\\x13\\xb5\\xddn\\xe9\\xe3K\\xf0E\\xb1]', '/home/user/test/tests/test_bytes.py::test_randbytes[\\x00i\\xb2\\x08\\x86\\x99\\x98"\\x1bw<\\xf4\\x84[\\xc4\\xb9\\x81\\x9f\\x00q\\xaes\\xce\\xfd\\xb4^\\xbe\\x12\\x8e\\xdcv\\xff|\\x06\\x9d\\x97]']
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/user/test
collected 2 items

============================ no tests ran in 0.01s =============================
ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[17]
(no name '/home/user/test/tests/test_bytes.py::test_randints[17]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[46]
(no name '/home/user/test/tests/test_bytes.py::test_randints[46]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[6]
(no name '/home/user/test/tests/test_bytes.py::test_randints[6]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[0]
(no name '/home/user/test/tests/test_bytes.py::test_randints[0]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[48]
(no name '/home/user/test/tests/test_bytes.py::test_randints[48]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[14]
(no name '/home/user/test/tests/test_bytes.py::test_randints[14]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[60]
(no name '/home/user/test/tests/test_bytes.py::test_randints[60]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randints[64]
(no name '/home/user/test/tests/test_bytes.py::test_randints[64]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x80\xde}=\x8c2/7\x00Y\x10`\x0b\xce%\xfd\xbd\xd0_\xee\x11\xe8rW\x8c\xb9\xf0N\xa11<f\xc9u\xb3\x92\x18\x0c\xeb\x99\x136?\x13\xc7\xda\xbe\xee\xc1t\x87\xeaE\xa7\xebk\x14\x86\xcdv]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x80\\xde}=\\x8c2/7\\x00Y\\x10`\\x0b\\xce%\\xfd\\xbd\\xd0_\\xee\\x11\\xe8rW\\x8c\\xb9\\xf0N\\xa11<f\\xc9u\\xb3\\x92\\x18\\x0c\\xeb\\x99\\x136?\\x13\\xc7\\xda\\xbe\\xee\\xc1t\\x87\\xeaE\\xa7\\xebk\\x14\\x86\\xcdv]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[c\xd9\xd9\x98|e`V\xc3\xce\x04N\xfc*\xfc)o\xb5\xa6\xe6\xedJ\x0eo\x94\xea\x17\xef]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[c\\xd9\\xd9\\x98|e`V\\xc3\\xce\\x04N\\xfc*\\xfc)o\\xb5\\xa6\\xe6\\xedJ\\x0eo\\x94\\xea\\x17\\xef]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x9b\xe4\xf63yd\xe1)\x02\x87\x17\xd3\xc4\xc7M\xacc\xab\x1cdJ\xca\x04\\x81\n\xca.\xe7\x9f\x13\xfd\xe3\x86\xa2c\xe2\xfe\xaf]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x9b\\xe4\\xf63yd\\xe1)\\x02\\x87\\x17\\xd3\\xc4\\xc7M\\xacc\\xab\\x1cdJ\\xca\\x04\\\\x81\\n\\xca.\\xe7\\x9f\\x13\\xfd\\xe3\\x86\\xa2c\\xe2\\xfe\\xaf]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x05\xd5\xed\x12\xa4!e\xaf\x16$\xfb"<\x91\xce\xd9\xb9\xdb`\xbe\xfd:\xf0\xd8/,\x84\xbdC\xf7\xdd\x0c\x94?\xe4\xe0\xbe]\xa2\xb53\xf6\xb80\x93\xfd\xbabC\xfc\xba\xe5s<\x00d]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x05\\xd5\\xed\\x12\\xa4!e\\xaf\\x16$\\xfb"<\\x91\\xce\\xd9\\xb9\\xdb`\\xbe\\xfd:\\xf0\\xd8/,\\x84\\xbdC\\xf7\\xdd\\x0c\\x94?\\xe4\\xe0\\xbe]\\xa2\\xb53\\xf6\\xb80\\x93\\xfd\\xbabC\\xfc\\xba\\xe5s<\\x00d]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xb8\x0e\xfcs\xa2]\xfe\xb2\xbe\x06\xd4\x83\x8e\x9a\xc1\xee\xd7K\nL=^\xb6\x99\xce1N\xf5\x06\x9f\xebM\xf0\xfb\x9c\xc0\x81\x04i\x10\x7f\xa9]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xb8\\x0e\\xfcs\\xa2]\\xfe\\xb2\\xbe\\x06\\xd4\\x83\\x8e\\x9a\\xc1\\xee\\xd7K\\nL=^\\xb6\\x99\\xce1N\\xf5\\x06\\x9f\\xebM\\xf0\\xfb\\x9c\\xc0\\x81\\x04i\\x10\\x7f\\xa9]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x92k\x11\x1c\xb0m\x89\x16j\x02"\xf3\xf2\xc9\xcb\x9aG]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x92k\\x11\\x1c\\xb0m\\x89\\x16j\\x02"\\xf3\\xf2\\xc9\\xcb\\x9aG]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xec"\nZ\xbc\xd5T\xb1\xeczX\xd1\xee]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xec"\\nZ\\xbc\\xd5T\\xb1\\xeczX\\xd1\\xee]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xe2\xa4\x84&\xb2[Hk\xc67\xaf\x13\x96\x18\xa6\xbd\x0bC\x1cS\xd1\xde\x9e\xfeJ\x87,\x00L\xefr\xc3\xfb-\x81\xdb\xf9\xc5]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xe2\\xa4\\x84&\\xb2[Hk\\xc67\\xaf\\x13\\x96\\x18\\xa6\\xbd\\x0bC\\x1cS\\xd1\\xde\\x9e\\xfeJ\\x87,\\x00L\\xefr\\xc3\\xfb-\\x81\\xdb\\xf9\\xc5]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\xdf\xab)u\x14\xe2\x8d\xe0\xd8\x13\xb5\xddn\xe9\xe3K\xf0E\xb1]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\xdf\\xab)u\\x14\\xe2\\x8d\\xe0\\xd8\\x13\\xb5\\xddn\\xe9\\xe3K\\xf0E\\xb1]' in any of [<Module tests/test_bytes.py>])

ERROR: not found: /home/user/test/tests/test_bytes.py::test_randbytes[\x00i\xb2\x08\x86\x99\x98"\x1bw<\xf4\x84[\xc4\xb9\x81\x9f\x00q\xaes\xce\xfd\xb4^\xbe\x12\x8e\xdcv\xff|\x06\x9d\x97]
(no name '/home/user/test/tests/test_bytes.py::test_randbytes[\\x00i\\xb2\\x08\\x86\\x99\\x98"\\x1bw<\\xf4\\x84[\\xc4\\xb9\\x81\\x9f\\x00q\\xaes\\xce\\xfd\\xb4^\\xbe\\x12\\x8e\\xdcv\\xff|\\x06\\x9d\\x97]' in any of [<Module tests/test_bytes.py>])

Finished running tests!
eleanorjboyd commented 10 months ago

ah I see now, we have been having some issues related to running pytest parameterized tests that generate different names for each run. The extension is looking for a given name but as the random number generates a new one, the name changes and it struggles to find what test to run. Im in the process of investigating, Ill update here with the results or another issue to reference. Thanks

eleanorjboyd commented 10 months ago

Hi! Gave this some investigation. It seems right now that running parameterize tests with renamed tests is not feasible. How we handle tests is via referencing all the test items and their IDs in order to handle UI like run, the pass or fail checkmarks etc. Without a stable id we can use to reference these items, we cannot run them from our UI. If the IDs were going to be the same random number every time it would work. So for example if you generate a set of random numbers, save these as constants then use that set of numbers (which will be stable between runs) for every run it will work but right now test ids which change between runs cannot be supported. Thanks