search

tarpas / pytest-testmon

Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
https://testmon.org
MIT License
800 stars 54 forks source link

Seeking examples of pytest-testmon in GitHub Actions #226

Open xianyuanliu opened 8 months ago

xianyuanliu commented 8 months ago

Is your feature request related to a problem? Please describe.

I'm looking to use pytest-testmon to optimize test execution time in the GitHub Action. I use the GitHub Action Cache to manage .testmondata storage and restoration. Here's my design approach:

  1. The main branch undergoes scheduled tests to generate a new .testmondata.
  2. Branches within PRs simply restore the generated .testmondata for a faster test, without updating it.

However, an issue arises when I execute pytest --testmon-nocollect within the PR: a new .testmondata is unexpectedly generated in the GitHub Action Cache. I'm unsure about the root cause of this. Could you possibly share any examples or repositories that have encountered and addressed a similar scenario?

Describe the solution you'd like When I run pytest --testmon-nocollect within the PR, no new .testmondata is generated in the GitHub Action Cache. However, executing pytest --testmon does result in the creation of a new .testmondata in the cache.

Describe alternatives you've considered Not yet.

Additional context Nope.

shristi0210 commented 7 months ago

Hello @xianyuanliu , I've been trying to cache the .testmondata like you mentioned in your issue. Would that be possible if you could help me with the code of doing so? Thanks.

xianyuanliu commented 7 months ago

I was unable to successfully resolve it yet due to differences between the cache systems of GitHub Action and the local machine. While I am glad to help, I might not be able to resolve the issue if it is the same issue.

tarpas commented 7 months ago

@xianyuanliu At first I misunderstood your suggested process. I think it makes more sense to:

  1. use www.testmon.net
  2. use pytest --tmnet --testmon-noselect on the main branch
  3. use pytest --tmnet on all other branches.

Collecting testmon data only nightly would lead to unnecessary re-execution of all tests related to all changes of recent changes and branches. As for "new" .testmondata being created I'm not sure what the main problematic aspect. --testmon-nocollect is intended for situation when you can't collect testmondata because you're running some incompatible coverage switches or you can't afford the overhead. I would hope none of that is your case. It's not intended to keep the .testmondata intact. Testmon is not designed that way.

Are you up for using --testmon and --testmon-noselect? Or should we explore more your use case of --testmon-nocollect?

marcodicro-dp commented 1 month ago

@tarpas Is there a way to run all my tests and build the testmon DB faster by running tests in parallel workers (e.g using a GHA matrix)?

Does --tmnet already handle that distributed processing case?