oblador / loki

👁 Visual Regression Testing for Storybook
https://loki.js.org/
MIT License
1.77k stars 173 forks source link

Add more info about `chromeTolerance` #113

Open vvscode opened 5 years ago

vvscode commented 5 years ago

I was looking about more info according to chromeTolerance and couldn't. At least range of possible values? Are we talking about 1-2-3-10 or about 0.01-0.2 and so on?

And please share reference for comparison option

Thanks in advance

oblador commented 5 years ago

Unfortunately this depends on the diffing tool used. I'd say it's more in the 0.01-0.2 range, but I generally recommend setting it to 0 unless you're having a good reason not to.

mowbell commented 5 years ago

@oblador Why reason currently does not passed pass --chromeTolerance option on react native simulators and emulators? I think is useful for emulators too, due that flags are runned by diffengine (gm-looks-same), and it does not depend of target (chrome simulator, etc) Currently on src/commands/test/run-tests.js createTargetTask does not pass any tolerance flag

const createTargetTask = configurations => {
    const { target } = configurations[Object.keys(configurations)[0]];
    switch (target) {
      case 'chrome.app': {
        return getTargetTasks(
          'Chrome (app)',
          createChromeAppTarget({
            baseUrl: options.reactUri,
            chromeFlags: options.chromeFlags,
          }),
          configurations,
          options.chromeConcurrency,
          options.chromeTolerance
        );
      }
      case 'chrome.docker': {
        return getTargetTasks(
          'Chrome (docker)',
          createChromeDockerTarget({
            baseUrl: options.reactUri,
            chromeDockerImage: options.chromeDockerImage,
            chromeFlags: options.chromeFlags,
          }),
          configurations,
          options.chromeConcurrency,
          options.chromeTolerance
        );
      }
      case 'ios.simulator': {
        return getTargetTasks(
          'iOS Simulator',
          createIOSSimulatorTarget(options.reactNativeUri),
          configurations,
          //options.chromeTolerance <--- IT's NOT PASSED
        );
      }
      case 'android.emulator': {
        return getTargetTasks(
          'Android Emulator',
          createAndroidEmulatorTarget(options.reactNativeUri),
          configurations,
          //options.chromeTolerance <--- IT's NOT PASSED
        );
      }
      default: {
        return die(`Unknown target "${target}`);
      }
    }
  };
abejfehr commented 2 years ago

I'm seeing this right now — there's minute differences around text rendering in React Native in CI and it's causing tests to fail but I don't see a way around this since I can't specify a tolerance

levymetal commented 2 years ago

@abejfehr you can pass options directly to the diffing engine by using the name of the engine as a key in your config. For example:

{
  "diffingEngine": "gm",
  "gm": {
    "tolerance": 0.00024
  }
}
techeverri commented 2 years ago

@abejfehr you can pass options directly to the diffing engine by using the name of the engine as a key in your config. For example:

{
  "diffingEngine": "gm",
  "gm": {
    "tolerance": 0.00024
  }
}

As @levymetal mentioned, you can pass options to the engine from the Loki config. Implemented by https://github.com/oblador/loki/pull/127 Docs: https://loki.js.org/configuration.html#diffingengine