NCAR / VAPOR

VAPOR is the Visualization and Analysis Platform for Ocean, Atmosphere, and Solar Researchers
https://www.vapor.ucar.edu/
BSD 3-Clause "New" or "Revised" License
177 stars 49 forks source link

Add comprehensive validation testing #3136

Open clyne opened 2 years ago

clyne commented 2 years ago

With the inclusion of a Python API, it is now possible to add automated, comprehensive validation testing to verify that renderers are behaving correctly with different data inputs. A validation system would exercise as many combinations of renderers and data sets as possible. Validation might be performed by comparing rendered results with "ground truth" using an image comparison such as SSIM. Ground truth images might be generated by a one-time, manual inspection of images with a known working version of the software.

clyne commented 2 years ago

We agreed to do this in two parts:

  1. A script that will input a config file (e.g. yaml) that specifies data sets, variables (2d and 3d), and a list of renderers to invoke, and will then output rendered images for each combination of data/variable/render
  2. An automated validation component that would use some kind of image comparison metric (e..g. SSIM) to compare "ground truth" with images generated by the script.

Part (1) is a 3.7 milestone. Part (2) will be deferred until later.

sgpearse commented 1 year ago

Closed with #3191.

clyne commented 1 year ago

Re-opening this but removing the milestone: (1) above is done, but (2) is not.

StasJ commented 4 months ago

This should also resolve #2986