search

SCons / scons

SCons - a software construction tool
http://scons.org
MIT License
2.1k stars 316 forks source link

RFE: adapt exsting test suite to pytest #4319

Open kloczek opened 1 year ago

kloczek commented 1 year ago

Describe the bug Looks like pyetst is failing because it cannot find TestCmd module.

Required information

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-scons-4.5.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-scons-4.5.1-2.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra -m 'not network' ============================= test session starts ============================== platform linux -- Python 3.8.16, pytest-7.2.2, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/scons-4.5.1 collected 0 items / 5 errors ==================================== ERRORS ==================================== ______________________ ERROR collecting src/test_files.py ______________________ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_files.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/test_files.py:38: in import TestSCons E ModuleNotFoundError: No module named 'TestSCons' ___________________ ERROR collecting src/test_interrupts.py ____________________ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_interrupts.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/test_interrupts.py:36: in import TestSCons E ModuleNotFoundError: No module named 'TestSCons' ____________________ ERROR collecting src/test_pychecker.py ____________________ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_pychecker.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/test_pychecker.py:33: in import TestSCons E ModuleNotFoundError: No module named 'TestSCons' ______________________ ERROR collecting src/test_setup.py ______________________ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_setup.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/test_setup.py:41: in import TestSCons E ModuleNotFoundError: No module named 'TestSCons' _____________________ ERROR collecting src/test_strings.py _____________________ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_strings.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) src/test_strings.py:39: in import TestCmd E ModuleNotFoundError: No module named 'TestCmd' =========================== short test summary info ============================ ERROR src/test_files.py ERROR src/test_interrupts.py ERROR src/test_pychecker.py ERROR src/test_setup.py ERROR src/test_strings.py !!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!! ============================== 5 errors in 1.11s =============================== ```

I see that module in the tree however look like it has location which does not allow use pytest

[tkloczko@pers-jacek scons-4.5.1]$ find . -name TestCmd\*
./testing/framework/TestCmd.py
./testing/framework/TestCmdTests.py

It woild be nice to have possibility to test scons using pytest.

bdbaddog commented 1 year ago

This is not a bug, but a feature request. SCons is meant to be tested by it's own framework.

If you'd like to submit a PR to enable using pytest.

mwichmann commented 1 year ago

On March 7, 2023 4:57:44 AM MST, "Tomasz Kłoczko" @.***> wrote:

Describe the bug Looks like pyetst is failing because it cannot find TestCmd module.

Not quite sure what you're after here. Yes, using pytest would be nice, but SCons has its own testing framework that extends unittest, and a very long history of that, it would be kind of hard to disentangle.

You need to look at the custom test runner, cleverly named runtest.py. It stuffs the path to the running SCons and the path to the framework (testing/framework) into PYTHONPATH, along with some other environment fiddling (things the framework is going to look at).

Also note scons unit tests don't follow the pytest naming convention, all the test files end in Tests.py (including the capital T). runtest has its own discovery mechanism.

kloczek commented 1 year ago

Adaptation to pytest wil involve main test sulte subtree reorganisation, It will require as well relative deep knowledge about existing test suite.

Nevertheless pytest is way more advanced tan unittest and I woild really encourage to move to pytest as it has tons of extensions which simplifies testing. Only from that point of view it may be worth to start thing about that by simple ctreate standard tests directory and start copy bit by bit test suite to that subtree. With such approach until would be still possible to use existing test suite.

bdbaddog commented 1 year ago

Honestly we're unlikely to get to this anytime soon. Our current test framework works well for our project needs.

We have many other priorities and limited resources.

As always, if you're up for taking this project on, please do. But, that said, whatever the update it would have to be a drop in replacement and not require overhauling too much of the existing tests. Because then we'd need to revalidate all those tests as well as the new test framework.

mwichmann commented 1 year ago

Don't disagree about the benefits of pytest, I've looked at this too. Have recently started introducing some of the specialized unittest asserts (test.assertEqual, etc.) because they provide more information without handcrafting the failure messages - but pytest has the smarts to do the same with just plain assert statements, i.e. you get details "for free".

As @bdbaddog says, this isn't likely to be any kind of high priority at this time. And the organization is relatively intentional - the original developers wanted to keep code/test/doc in proximity, so there is often a pattern of (SCons/Module.py, SCons/ModuleTests.py, SCons/Module.xml), rather than the current Python project recommendation of having those each in their own directory tree. It might take some convincing to move away from that, though nothing is impossible...

kloczek commented 1 year ago

Whatabout diffrent approach by use just pytest temporary to call existing test sute? 🤔

This woild allow to reuse 100% existing test suite and add completly new usinits only in pytest ..

bdbaddog commented 1 year ago

Let me reiterate: "We have many other priorities and limited resources." This is not really anywhere in the priority list currently.

mwichmann commented 1 year ago

For grins, I ran the existing unittests (which are the minority of tests in the entire thing we call the test suite. It was relatively easy to do: you need to pass testing/framework in PYTHONPATH, and you need to change the test discovery since we name unittest files differently. An entry in pytest.ini was enough to do that:

[pytest]
python_files = *Tests.py

The results were: the tests ran, but ran fairly poorly. Had hoped for maybe a bit better, since pytest knows how to run unittest-based tests. It seems pytest is a lot fussier about scope, and so many things that are imported or declared in the module global scope and work in unittest are not visible in parts of the testing under pytest.

====== 594 failed, 347 passed, 8 skipped, 3 warnings, 15 errors in 15.85s ======