search

pytest-dev / pytest-repeat

pytest plugin for repeating test execution
https://pypi.python.org/pypi/pytest-repeat/
Other
169 stars 27 forks source link

repeating a test spams pytest cache #84

Open dpinol opened 5 months ago

dpinol commented 5 months ago

When running many repeatitions of a test with --count, .pytest_cache/v/cache/nodeids may get up to millions of entries. Eg:

"test/utils/test_types.py::test_if_none[1-100]",
  "test/utils/test_types.py::test_if_none[10-100]",
  "test/utils/test_types.py::test_if_none[100-100]",
  "test/utils/test_types.py::test_if_none[11-100]",
  "test/utils/test_types.py::test_if_none[12-100]",
...

This may cause a lag of several seconds even after executing a single method with -k specifying a single test file.

The culprit is pytest_sessionfinish in cacheprovider.py, when calling config.cache.set("cache/nodeids", sorted(self.cached_nodeids)). When cached_nodeids contains milions of entries, this function always generates a json with the whole contents of .pytest_cache/v/cache/nodeids.

RonnyPfannschmidt commented 5 months ago

Please elaborate on the actual numbers

dpinol commented 5 months ago

The length of ".pytest_cache/v/cache/nodeids" depends on the usage. I guess I executed at some time with millions of executions and got the file with so many lines. In that case, on my intel i9 a trivial test took 4s, while after removing the file the whole execution now takes 0.3s

dpinol commented 5 months ago

I tried to reproduce it. I executed 1M times a file with 2 tests. It created a cache file of 143M with 2M entries. Executing the tests a single time passed from 0.4s to 2.4s

RonnyPfannschmidt commented 5 months ago

as repeat uses parameterize - i suspect a breaking change is needed to make this more usable

okken commented 4 months ago

You can disable caching entirely while running large counts with -p no:cacheprovider.
Would that work?