search

InsightSoftwareConsortium / ITK

Insight Toolkit (ITK) -- Official Repository. ITK builds on a proven, spatially-oriented architecture for processing, segmentation, and registration of scientific images in two, three, or more dimensions.
https://itk.org
Apache License 2.0
1.37k stars 660 forks source link

Tolerance in image size in regression comparison #4613

Open dzenanz opened 2 months ago

dzenanz commented 2 months ago

Description

Currently, the test image and its regression baseline must have the exact same size, as per: https://github.com/InsightSoftwareConsortium/ITK/blob/e41c4583c03c45fda7334111041aabb4b74e8465/Modules/Core/TestKernel/src/itkTestDriverInclude.cxx#L508-L514

Which causes this sort of failure:

The size of the Baseline image and Test image do not match!
Baseline image: C:/Dev/ITK-vs22/ExternalData/Modules/Remote/Montage/test/Input/MNML_5_500x_101/MNML_5-ITK.png has size [7447, 8346, 1, 1, 1, 1]
Test image:     C:/Dev/ITK-vs22/Testing/Temporary/itkMontageGTMNML5_2_1.mha has size [7447, 8345, 1, 1, 1, 1]

This makes the regression test fail even if it would pass otherwise (with e.g. --compareRadiusTolerance 1 and other tolerances).

Expected behavior

Use compareRadiusTolerance for comparing image sizes (which would have backwards compatibility problems), or use a new flag for that purpose (which would not disturb existing comparisons).

Actual behavior

There is no way to specify that image sizes could differ even a little.

dzenanz commented 2 months ago

In implementing this, the images could be cropped to the common region, or expanded to the largest containing region. For expansion, we would need to decide how to fill the padding. Perhaps go for the most convenient NearestNeighborExtrapolateImageFunction?

blowekamp commented 2 months ago

The current implementation assumes that the image are "congruent"? Aside from a different size, would this proposal allow for any other variations in geometry metadata? Are we comparing values physical or pixel space?

P.S.: If we are comparing image A and B, then a perhaps new flexible comparison option is to resample image B onto the grid of image A, then proceed with the existing comparison. This essentially remove the requirement for image B to be congruent with image A.

dzenanz commented 2 months ago

For ITKMontage's tests, the origin is always the same, but size might differ. So change for this case would be relatively simple.

For the resampling approach, we should probably do it symmetrically, so both resample A to B, and B to A, and do existing comparison twice (in case one image a significantly bigger than the other). And do this only if the images are not congruent and the size tolerance was specified, otherwise do the classic comparison as before.

Are you willing to take this up this issue Brad?

dzenanz commented 2 months ago

It would be nice to clean up this flag in Montage: InsightSoftwareConsortium/ITKMontage@06cacd6 (#226)