search

pydata / pydata-sphinx-theme

A clean, three-column Sphinx theme with Bootstrap for the PyData community
https://pydata-sphinx-theme.readthedocs.io
BSD 3-Clause "New" or "Revised" License
615 stars 319 forks source link

Benchmark #863

Open jarrodmillman opened 2 years ago

jarrodmillman commented 2 years ago

To more quickly identify performance regressions we should add some benchmarking tests. This has been suggested before (see https://github.com/pydata/pydata-sphinx-theme/issues/381#issuecomment-817981156 for example). And maybe we could have caught #855 earlier in the release process for 0.10.

We could build a packages documentation (e.g., NetworkX since it is pure Python) with the main branch every week or so (if there have been any commits since the last run of the benchmarking suite). @MridulS has been looking into benchmarking for NetworkX, so he may have some input (or may be willing to help).

choldgraf commented 2 years ago

Another approach we could take is to check the duration time of one of our pytests (https://howchoo.com/python/how-to-measure-unit-test-execution-times-in-pytest) and the fail if it exceeds some threshold. Kind of like a time-based regression test.

drammock commented 2 years ago

We have something like that in MNE, I can implement if you want. If you can point to which test(s) you want timed that would help.

choldgraf commented 2 years ago

I think this one is the first and simplest build:

https://github.com/pydata/pydata-sphinx-theme/blob/fb0a4e7764d34bc7928c0dc3e91c9a5aa138ff1b/tests/test_build.py#L55

So maybe after running our test suite, we just run that one like 5 times, take the median, and then check it hasn't exceeded some amount?