[Feature Request] Allow a MaxPassPercentage when testing goldens

[bw-flagship]

In my golden tests, I oftentimes have a failure but a diff from 0.00%. This happens e.g. because the macOS version of my build_runner is slightly different from my local machine. I would appreciate it if I could configure the golden_toolkit in a way that it sees 99.99% accuracy as a successful test.

[quoc-huynh-cosee]

You can achieve this by creating a flutter_test_config.dart file with a custom LocalFileComparator. Here is a LocalFileComparator with a threshold:

class LocalFileComparatorWithThreshold extends LocalFileComparator {
  LocalFileComparatorWithThreshold(super.testFile, this.threshold)
      : assert(
          threshold >= 0 && threshold <= 1,
          'Threshold must be between 0 and 1! ',
        );

  final double threshold;

  @override
  Future<bool> compare(Uint8List imageBytes, Uri golden) async {
    final result = await GoldenFileComparator.compareLists(
      imageBytes,
      await getGoldenBytes(golden),
    );

    if (!result.passed && result.diffPercent <= threshold) {
      debugPrint(
        'A difference of ${result.diffPercent * 100}% was found, but it is '
        'acceptable since it is not greater than the threshold of '
        '${threshold * 100}%.',
      );

      return true;
    }

    if (!result.passed) {
      final error = await generateFailureOutput(result, golden, basedir);
      throw FlutterError(error);
    }

    return result.passed;
  }
}

Here you can override the GoldenToolkit configuration:

Future<void> testExecutable(FutureOr<void> Function() testMain) async {
  return GoldenToolkit.runWithConfiguration(
    () async {
      if (goldenFileComparator is LocalFileComparator) {
        final testUrl = (goldenFileComparator as LocalFileComparator).basedir;

        goldenFileComparator = LocalFileComparatorWithThreshold(
          Uri.parse('$testUrl/test.dart'),
          0.01, // Set the threshhold here e.g. 1%
        );
      } else {
        throw Exception(
          'Expected `goldenFileComparator` to be of type '
          '`LocalFileComparator`, but it is of type '
          '`${goldenFileComparator.runtimeType}`.',
        );
      }
      await loadAppFonts();
      return testMain();
    },
    config: GoldenToolkitConfiguration(),
  );
}

Important to mention is that the tests will pass, but it won't prevent generating new images.

[bw-flagship]

Great, thanks a lot!