Open banesullivan opened 1 year ago
IIRC our testing was too slow when we actually rendered everything, so we hacked our tests to SetVisibility(False)
on almost all visuals. We could add an env var to control this on main
, though, and then it would produce correct images...
our testing was too slow when we actually rendered everything
Oh... oof, it may be good to run this before releases (for PyVsita and/or MNE) but that slowdown would probably prove problematic then
@larsoner Could this be tested on the rendered docs then?
PyVista does provide a pyvista.compare_images
method which I recently used to compare different versions of built documentation images. My very rough script was:
Yes in principle, but that would be way slower than just reenabling visibility in our unit tests :)
Looking into it on my laptop, time pytest mne/viz/_brain
(which should be the slowest part) takes 94 sec on main
. If I tweak the tests to make mne/viz/backends/_pyvista.py:_hide_testing_actor
just return
instead of hiding the actors, it actually takes the same amount of time. This isn't a fair representation of what will happen on CIs, though, because I have a proper GPU. It might be worth trying a monkey-patch of this function in PyVista to see how long it takes CIs there. @banesullivan I could try this if you want. If it's promising and we want to keep it, we could add a _MNE_TESTING_SHOW_ACTORS
env var to do this short-circuiting...
Describe the new feature or enhancement
PyVista has contract testing with MNE to make sure we don't break anything you all are using. I'm wondering if we want to take this a step further with image regression testing to validate further we aren't changing the visualizations you all create here
AFAIK, that isn't implemented here, but do let me know if I'm not correct.
Describe your proposed implementation
Now that
pytest-pyvista
is implemented, this should be straightforward to set up for the visualization tests. Though... I looked at the tests here, and it wasn't immediately obvious to me where/how PyVista is used in the testshttps://github.com/pyvista/pytest-pyvista
Describe possible alternatives
Or just leave it as is, which only validates API usage and not actual rendering results.
Additional context
No response