Unpredictable test output - v1.5

[simondib]

I am at a loss to work out what's going with these test results:

Test 1 should have failed Test 2 & 3 failed as expected

Test 1 & Test 3 have test outputs are both blank (for reasons unknown), but only Test 3 has failed as expected.

I am running the test suites against a local browsersync server (Webpack/Gulp). Test pages inside the browser all appear as expected.

Any ideas?

[NimaSoroush]

Hi @simondib: Currently differencify-jest-reporter only looks for automatic test name generated by jest. If you have custom test name the reporter is not able to find the path for those tests. That is something need to be updated in next version of differencify-jest-reporter

Are your test names custom names?

[NimaSoroush]

Also when you run them, do you see that __current_output__ folder being generated with all your snapshots in it?

Please paste your jest reporter config here as well

[simondib]

Reporter config:

 "reporters": [
      "default",
      [
        "differencify-jest-reporter",
        {
          "debug": false,
          "reportPath": "differencify_reports",
          "reportTypes": {
            "html": "index.html",
            "json": "index.json"
          }
        }
      ]
    ],

Current output is being generated. Sample output filename:

Fixture Match - Club - PREGAME appear correctly on small devices.current.png

Generated from this test:

describe('Fixture Match - Club - PREGAME', () => {
  beforeAll(async () => {
    jest.setTimeout(CONFIG.global.timeout);
  });

  afterAll(async () => {
    await differencify.close();
  });

  it('appear correctly on small devices', async () => {
    await differencify
      .init()
      .launch(CONFIG.launch)
      .newPage()

      .setViewport(getViewport('small', VIEWPORT_HEIGHT))
      .goto(TEST_HOST + PATH)
      .waitFor(4000)

      .screenshot()
      .toMatchSnapshot()
      .close()
      .end();
  });
});

[NimaSoroush]

I wonder why your test name from the differencify report picture you posted above is different from current image name. I expect to see

testname: Fixture Match - Club - PREGAME appear correctly on small devices

reference snapshot file: Fixture Match - Club - PREGAME appear correctly on small devices.png

current snapshot: Fixture Match - Club - PREGAME appear correctly on small devices.current.png

differencified snapshot: Fixture Match - Club - PREGAME appear correctly on small devices.differencified.png

Can you confirm you see all of them in case of failure?

[simondib]

@NimaSoroush my apologies, I provided the wrong filename/test harness in my post above. However the problem with false positives persists in other tests. Here's one I've just run now:

"Pass" when it clearly should be a fail. Also reported as "pass" in the console.

Testname: Sponsor logos appear correctly on small devices

Current snapshot: Sponsor logos appear correctly on small devices.current.png

Reference snapshot: Sponsor logos appear correctly on small devices.snap.png

Differencified snapshot: Sponsor logos appear correctly on small devices.differencified.png

[simondib]

@NimaSoroush could this be as simple as the current image not being properly deleted prior to the subsequent test report being generated? The failed image above is from the previous test - it shouldn't appear in the report of a passed test.

[NimaSoroush]

Yeah. That seems to be the issue. I can put a fix for it or if you need this quick feel free to raise a PR

[NimaSoroush]

I fixed the issue here https://github.com/NimaSoroush/differencify/pull/105

v1.5.1 should be out and ready to use https://www.npmjs.com/package/differencify