NimaSoroush / differencify

Differencify is a library for visual regression testing
MIT License
634 stars 46 forks source link

Tests pass even when error occurs in tests #147

Open glentakahashi opened 5 years ago

glentakahashi commented 5 years ago

Right now if you experience an error with puppeteer or the code itself, the tests still pass. E.g.: (this specific issue of slashes in test names is fixed, but it still fails on other issues as well)


  console.log node_modules/differencify/dist/utils/logger.js:64
    14:43:32 tests that nginx functions the same as app for /mypage: Executing page.toMatchSnapshot step

  console.log node_modules/differencify/dist/utils/logger.js:64
    14:43:32 tests that nginx functions the same as app for /mypage: screenshot saved in -> /test/__image_snapshots__/tests that nginx functions the same as app for /mypage.snap.png

  console.error console.js:188
    Trace: 14:43:32 tests that nginx functions the same as app for /mypage: { Error: ENOENT: no such file or directory, open '/test/__image_snapshots__/tests that nginx functions the same as app for /mypage.snap.png'
        at Object.fs.openSync (fs.js:646:18)
        at Object.fs.writeFileSync (fs.js:1299:33)
        at writeFileSync (/test/node_modules/differencify/src/compareImage.js:124:6)
        at Target._evaluateResult (/test/node_modules/differencify/src/target.js:255:29)
        at Target._evaluateResult (/test/node_modules/differencify/src/target.js:242:31)
        at handleAsyncFunc (/test/node_modules/differencify/src/helpers/functions.js:2:67)
        at Target._handleFunc (/test/node_modules/differencify/src/target.js:99:45)
        at ChainObject.end (/test/node_modules/differencify/src/helpers/proxyChain.js:40:24)
        at <anonymous>
        at process._tickCallback (internal/process/next_tick.js:189:7)
      errno: -2,
      code: 'ENOENT',
      syscall: 'open',
      path: '/test/__image_snapshots__/tests that nginx functions the same as app for /mypage.snap.png' }
        at Logger.trace (/test/node_modules/differencify/src/utils/logger.js:66:15)
        at PrefixedLogger.trace (/test/node_modules/differencify/src/utils/logger.js:23:17)
        at Target.trace [as _logError] (/test/node_modules/differencify/src/target.js:27:25)
        at Target._logError [as _evaluateResult] (/test/node_modules/differencify/src/target.js:257:14)
        at <anonymous>
        at process._tickCallback (internal/process/next_tick.js:189:7)

  console.log node_modules/differencify/dist/utils/logger.js:64
    14:43:32 Closing browser...

 RUNS  ...

 PASS  ./test.js (205.754s)```
NimaSoroush commented 5 years ago

Hi @glentakahashi , do you have an example of your test structure which you can post here?