search

tarpas / pytest-testmon

Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
https://testmon.org
MIT License
823 stars 55 forks source link

Using `pytest --testmon --cov` to see merged coverage results #176

Closed brendanator closed 1 year ago

brendanator commented 2 years ago

When I run pytest coverage along with testmon the coverage reports only the code that was run in the selected tests.

It would be incredible if the coverage reported the merged data from the tests that are selected and the rest of the data in .testmondata.

I'd love to get instantly updated test converage over a whole project. Is this possible?

tarpas commented 2 years ago

That is not unthinkable but quite difficult. Coverage data refers to the source code by line numbers, so it becomes invalid whenever you add or remove a line. I would say it's out of scope of this project for now.

landeholt commented 2 years ago

That is not unthinkable but quite difficult. Coverage data refers to the source code by line numbers, so it becomes invalid whenever you add or remove a line. I would say it's out of scope of this project for now.

Are there no other ways around this? I've just recently found your work @tarpas - so I am quite green when it comes to your code. But I've had a few hours of tinkering with it on my local machine.

What is exactly needed in order to make this work?

I've noticed that you erase the coverage reports for each test in /tests - why is that? Can't we just accumulate that json data?

EDIT:

wouldn't it just be an instance of caching and upserting changed or added "morfs"?

landeholt commented 2 years ago

Seems like commenting out this line

https://github.com/tarpas/pytest-testmon/blob/67c80541367752cb671fb68db990011dec5de1d3/testmon/testmon_core.py#L346

doesn't seem to affect testmon, but allows coverage to keep track on the coverage data, and it looks like coverage is upserting their filenames: coveragepy/sqldata.py

meaning that it (should) accomplish what @brendanator wished for.

EDIT:

I did later on notice changes in testmon. So removing self.cov.erase()shouldn't be done. 😞

tarpas commented 2 years ago

As I wrote:

Coverage data refers to the source code by line numbers, so it becomes invalid whenever you add or remove a line.

Also to have complete picture: .testmondata is a "lossy" format. Only method checksums are stored.

Testmon would have to come up with "lossless" .testmondata format which is a challenge by itself (plus keep it performant and not eating too much space)

tarpas commented 1 year ago

This has been implemented and release to the extent it's technically practical/feasible. The results are not perfect, see discussion here: https://github.com/tarpas/pytest-testmon/issues/86