search

tarpas / pytest-testmon

Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
https://testmon.org
MIT License
823 stars 55 forks source link

Examples of high-profile projects using `pytest-testmon`? #171

Open fkiraly opened 3 years ago

fkiraly commented 3 years ago

At sktime, we are looking for ways to speed up our CI testing workflows.

Incremental testing might be an option, see https://github.com/alan-turing-institute/sktime/issues/1408

Now it doesn't seem too wide-spread, at least your documentation does not seem to point to projects that would be using this in CI.

Are you aware of any examples? If yes, would it make sense to point to them in your docs? Does it even make sense to use pytest-testmon in CI?

fkiraly commented 2 years ago

?

tarpas commented 2 years ago

Hi Franz, yes it does make sense to use pytest-testmon in CI, however you would need to implement a couple of extra pieces yourself. One example of usage from longer ago is here: https://kodare.net/2017/10/24/vastly-faster-python-integration-tests.html

I do plan to implement a very convenient and automatic sharing server and I'm looking for a good project with which to cooperate on this.

How good does testmon work for you locally now? How deterministic and independent are the tests? Could you try suggestions from https://testmon.org/hidden-test-dependencies.html (running each test in separate process and running all tests in reverse)?

fkiraly commented 2 years ago

Hm, I'm not sure whether I understand what you mean with "sharing server" here.

Does testmon require a separate server to run? That sounds strange, since I have naively expected that you can run a coverage tool to see which tests touch which lines of code.

How good does testmon work for you locally now?

I haven't really tried it, locally I run tests selectively. Further, tests in sktime should be deterministic and independent - but there are a few instances of stochasticity which come from unresolved but known bugs.

tarpas commented 2 years ago

You can run a coverage tool to see which tests touch which lines of code. But by default CI discards all state information across builds which prevents incremental testing.

To achieve anything incremental, you need to share data across builds. You can do that with your own scripts as Anders described in the article I gave. Or you can wait until we implement some simple solution as part of testmon, which I'm quite sure would include some client/server.

In https://github.com/alan-turing-institute/sktime/issues/1408 you write that you would like to to use this for PRs. That' exactly what I would do :).

I'll be very busy coming 2 weeks so nothing much will happen here. But I'm quite motivated to work on it later because it's a good idea which more and more people are asking for.

fkiraly commented 2 years ago

You can do that with your own scripts as Anders described in the article I gave.

I think at sktime this is not a priority, we are quite busy with the package itself...

Or you can wait until we implement some simple solution as part of testmon, which I'm quite sure would include some client/server.

Hm, would that work similar to codecov? If so, is there maybe a smart way to use the codecov data directly? Alternatively, could the test coverage reports and diffs not be stored in the repository, or a separate special purpose repository? They will be of interest only for the main branch (or more generally CI target branches).

With the "dedicated server" solution, I'm worried that it may go the path of so many such server solutions, i.e., if it's too niche, the plug may get pulled at some random time when maintenance disintegrates; if it's too successful, the plug may get pulled through commercialization or paywalls in the vein of Anaconda or Docker Desktop.

tarpas commented 2 years ago

I think at sktime this is not a priority, we are quite busy with the package itself...

Sure, I'm getting to a point where I could have time for this.

Or you can wait until we implement some simple solution as part of testmon, which I'm quite sure would include some client/server.

Hm, would that work similar to codecov? If so, is there maybe a smart way to use the codecov data directly? Alternatively, could the test coverage reports and diffs not be stored in the repository, or a separate special purpose repository? They will be of interest only for the main branch (or more generally CI target branches).

With the "dedicated server" solution, I'm worried that it may go the path of so many such server solutions, i.e., if it's too niche, the plug may get pulled at some random time when maintenance disintegrates; if it's too successful, the plug may get pulled through commercialization or paywalls in the vein of Anaconda or Docker Desktop.

I understand your concern. My strategy is to validate this technically using the least amount of coding possible first. (That's probably using AWS Lambda and RDS/DynamoDB). Then let's hope it's successful and can be sustained to provide free service to opensource projects like GitHub and GitHub Actions :-).

tarpas commented 2 years ago

Hi Franz. We're working on this and it's not difficult to set-up using github actions cache. However when trying with your project I realized we don't have support doctest modules support.

fkiraly commented 2 years ago

Hi Franz. We're working on this and it's not difficult to set-up using github actions cache. However when trying with your project I realized we don't have support doctest modules support.

Excellent!

Perhaps we can just leave doctest out for the moment and construct a GitHub actions based solution? If you let me know where we could help, and there's a concrete technical plan, we could look into helping with doctest.