search

pyvista / pytest-pyvista

Plugin to test PyVista plot outputs
https://pytest.pyvista.org
MIT License
13 stars 5 forks source link

Dump failing images to a directory #6

Closed banesullivan closed 1 year ago

banesullivan commented 2 years ago

I'm dealing with a scenario right now where a lot of tests are failing the image regression test on Windows but not other platforms on CI. It would be nice if there were an option to dump the failing images to a specific path so that I can upload those as a CI artifact to inspect. I'm currently manually doing this

banesullivan commented 2 years ago

I am doing this in https://github.com/pyvista/pyvista/pull/3609 which would need to migrate here

MatthewFlamm commented 2 years ago

Another option is to save all image files during a test to a folder (default to a temporary folder). Then you could upload this to an artifact in CI. This would lead to larger artifacts, but also has other advantages, for example being able to inspect for false negatives. I don't have a feeling one way or the other.

banesullivan commented 2 years ago

Good point. I'd rather this be to save all produced images, not just failing ones

MatthewFlamm commented 1 year ago

To expand upon this: I think we should have two configurable directories. 1) the stored image cache, 2) the generated images for a single run.

1) stored image cache. Can there be a universal way to set this? It should be related to the testing code, which could be hierarchical, so tricky to access here. If it is required to be explicitly set by the user, we should check that this is done, or raise an error. 2) generated images. This could be in the shared appdirs by default.

It would also be great in the future to allow for an optional check that all the files in the cache dir have a corresponding file in generated dir, and vice versa. This would ensure that everything is working as expected during testing.