The default Parameter's in screen shot testing are strict. This means that tests will be flaky.
For example I am using in my add-on testing the following settings
// We do screenshot testing, adjust settings to ensure less flakiness
Parameters.setScreenshotComparisonTolerance(0.05);
Parameters.setScreenshotComparisonCursorDetection(true);
testBench().resizeViewPortTo(800, 600);
Parameters.setMaxScreenshotRetries(3);
Parameters.setScreenshotRetryDelay(1000);
The default view port of app testing should be larger than 800 x 600 I use for add-ons. Point here is that we should ensure some fixed default size, since otherwise the view port size can wary.
More over I sometimes run tests also in Development mode, when dev mode gizmo needs to be hidden
// Hide dev mode gizmo, it would interfere screenshot tests
try {
$("vaadin-dev-tools").first().setProperty("hidden", true);
} catch (NotFoundException e) {
}
Screenshot testing would be more approachable if the defaults would be something like this. Now developers need to consume time on trial and error to find these.
The default Parameter's in screen shot testing are strict. This means that tests will be flaky.
For example I am using in my add-on testing the following settings
The default view port of app testing should be larger than 800 x 600 I use for add-ons. Point here is that we should ensure some fixed default size, since otherwise the view port size can wary.
More over I sometimes run tests also in Development mode, when dev mode gizmo needs to be hidden
Screenshot testing would be more approachable if the defaults would be something like this. Now developers need to consume time on trial and error to find these.