zinserjan / wdio-visual-regression-service

Visual regression service for WebdriverIO.
MIT License
102 stars 39 forks source link

Auto reference screenshot generation foils testing in retry scenario #105

Open alisman opened 2 years ago

alisman commented 2 years ago

Retries are an essential part of e2e in order to deal with flakey tests.

When a screenshot is missing, the framework creates a new one and saves it. This causes a retry to PASS. I would argue that the framework should never automatically create a reference screenshot. Or at least there should be a flag to prevent if from doing so. The reference screenshots should be copied manually from screens directory by developers.