python / cpython

The Python programming language
https://www.python.org
Other
63.73k stars 30.53k forks source link

gh-127221: Add colour to unittest output #127223

Open hugovk opened 5 days ago

hugovk commented 5 days ago

Use the existing _colorize module already used for tracebacks and doctest so output can be controlled with the PYTHON_COLORS, NO_COLOR and FORCE_COLOR environment variables.

This PR does a couple of things to add colour:

Tested with some small demo scripts:

No tests ran

run-unittests-none.py ```python import unittest if __name__ == "__main__": unittest.main() ```
Before After pytest
image image image

Passed

run-unittests-pass.py ```python ```
Mode Before After pytest
Regular image image image
Verbose image image image

Failing

run-unittests.py ```python import unittest class TestThings(unittest.TestCase): def test_pass(self): self.assertTrue(True) def test_fail(self): self.assertTrue(False) def test_error(self): raise Exception("Manually raised exception") def test_pass2(self): self.assertEqual(1, 1) @unittest.skip("demonstrating skipping") def test_skip(self): self.fail("shouldn't happen") class ExpectedFailureTestCase(unittest.TestCase): @unittest.expectedFailure def test_fail(self): self.assertEqual(1, 0, "broken") @unittest.expectedFailure def test_pass(self): self.assertEqual(1, 1, "not broken") if __name__ == "__main__": unittest.main() ```
Mode Before After pytest
Regular image image image
Verbose image image image

Subtests

run-unittests-subtests.py ```python import unittest class DoSubTest(unittest.TestCase): def test_subtest(self): """Test with subtest""" for i in range(0, 4): with self.subTest(i=i): if i == 2: raise Exception("Manually raised exception") self.assertEqual(i % 2, 0) if __name__ == "__main__": unittest.main() ```
Mode Before After pytest
Regular image image image
Verbose image image image

πŸ“š Documentation preview πŸ“š: https://cpython-previews--127223.org.readthedocs.build/

nineteendo commented 5 days ago

Looks good, I see you're taking some inspiration from pytest, but I had to zoom in all the way to be able to read the text. :)

hugovk commented 5 days ago

You can click the images to open them full size.

nineteendo commented 5 days ago

Well, I wanted to compare the images so I had to zoom in to see the difference. Maybe we could only show the comparison between pytest? The uncoloured images don't add much value.

hugovk commented 4 days ago

This is great!! With the coloured tracebacks as well, though, I sort-of feel like there might be a bit too much colour now when tests fail. It feels slightly overwhelming!

What about making only the "ERROR"/"FAIL"/"UNEXPECTED SUCCESS" lines coloured red, and not the ======= line above or the ------- below?

Thanks! Here's without coloured separators:

image

And verbose:

image

What do you think?

Also these demo scripts are a bit contrived; hopefully most of the time, most of your tests pass and just one or two fail. But then again, sometimes a lot do fail!

AlexWaygood commented 4 days ago

Nice, that looks great to me now!

nineteendo commented 4 days ago

How about swapping the rows and columns in the table? It makes the text better readable:

show table
Mode Regular Verbose
Before image image
After image image
pytest image image
AlexWaygood commented 4 days ago

How about swapping the rows and columns in the table? It makes the text better readable:

To be clear @nineteendo, this is a comment on the table in Hugo's PR description rather than the code he's changing in this PR, correct?

nineteendo commented 4 days ago

Correct, with 3 images on the same row I can't read the text without zooming in.

AlexWaygood commented 4 days ago

I see.

@hugovk, do you think we should add a link to the using-on-controlling-color somewhere in the unittest docs? I see we didn't when we added color support to doctest, but maybe we should have done

hugovk commented 4 days ago

@hugovk, do you think we should add a link to the using-on-controlling-color somewhere in the unittest docs? I see we didn't when we added color support to doctest, but maybe we should have done

Yeah, good idea. Something like https://github.com/python/cpython/pull/127223/commits/5a1de7cbf9bde44e74534fb92b8444b9d95c6783?

Let's also add it to doctest and traceback.

hugovk commented 4 days ago

Thank you! I like the idea. But, I personally feel like it is still a bit too colorful. Right now almost nothing is white.

Almost nothing is white because these demo scripts are intentionally crafted to generate lots of failures :) It's very similar to pytest.

One more idea: can you please test this with the white theme as well?

Sure:

unittest pytest
image image
image image
image image
image image
image image

Note I've not adjusted my local colours for this light theme. If I was using it normally, I may have adjusted it already for pytest.