Test case specific coverage reports

[villesau]

🚀 Feature Proposal

Jest should support test case specific coverage reporting.

Motivation

When taking code coverage test by test basis, it would enable writing a reporter that sorts failed tests based on the blast radius: If you have multiple failing tests, the one with smallest footprint/least lines affected/least functions called is likely the test you should focus on fixing since that's closest to the failing code. By fixing the simplest test case, you are more likely to fix other tests as well. Or at least make sure that other failed tests does not fail to same issue, which is also valuable.

Modern testing often tests many layers from very small units to full blown api tests. When you break something fundamental, you might have single unit test covering it, but also 15 api tests start failing because of the same functionality. In this case you should not focus on the API tests, but the one with smallest blast radius since it is likely to fix the rest of the tests.

If you focus on test that has larger blast radius, there's greater risk that this test covers multiple erroneous cases. Smaller test case has smaller risk for this. It would also give more value to write smaller tests since they would highlight failed code much better than larger tests.

Example

I have a whole POC written for this: https://github.com/villesau/test-fix-ordering-poc

In the above example the failed tests are sorted in a way that the smallest broken unit is always on top. As you see, every time you fix the smallest unit, also some other tests get fixed for free. That way you always find the fastest possible order to fis your tests. This saves you from a lot of hassle, especially with code bases you are not yet familiar with.

You can see my jest changes here (note, very hacky just to get the PoC working): here

Pitch

Why does this feature belong in the Jest core platform?

This belongs to Jest since there is no way to write plugin for taking test case specific coverage reports. It would also provide much greater visibility over which failed test to focus on. In overall this would be a significant productivity boost. Test runner should be able to point out which failed test to focus on and this is the way to make it possible. This has potential to change the way we think about testing and testing pyramids.

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[villesau]

still valid.

[SimenB]

Oh, I like this idea! I don't think we should enable the inspector and collect coverage by default, but behind some config/cli flag might make sense 🙂

[villesau]

@SimenB if you are interested, see my PoC repos! :) I got it working back in the days, but I'm fairly sure there are better ways to achieve it, I just hacked it in the first place where I see it is possible. I also do not know how (test case specific) coverage reports would affect performance. Maybe the CLI could have option to "sort by blast radius" along with "run failed only" etc. which would enable coverage under the hood if it has considerable effect on performance?

[villesau]

I think the bigger the codebase grows, the more value this kind of functionality could bring.

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[villesau]

I still think this would be a good idea :+1:

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[villesau]

Still thinking this is an awesome idea.