process fails on javascript errors without an error exit code

shlomitc commented 5 years ago

The scenario

I had the following test where undefinedStoryName was accidentally used but not defined.

Reproducable code:

import React from 'react';
import { storiesOf } from '@storybook/react';

storiesOf(`some test`, module).add(undefinedStoryName, () => (
  <div style={{ height: '500px', color: 'white' }} />
));

Expected behavior:

eyes-storybook fails with exit code 1.

Actual behavior:

> eyes-storybook

Using @applitools/eyes-storybook version 2.6.25.

✔ Storybook was started
⠦ Reading stories
Test is finished but no results returned.

More info:

running with showLogs outputs the hidden error:

019-06-06T21:15:23Z Eyes: [LOG    ] {} (): eyesStorybook started
2019-06-06T21:15:23Z Eyes: [LOG    ] {} (): browser launched
2019-06-06T21:15:23Z Eyes: [LOG    ] {} (): 3 pages open
2019-06-06T21:15:23Z Eyes: [LOG    ] {} (): got script for processPage
2019-06-06T21:15:23Z Eyes: [LOG    ] {} (): finished creating functions
⠋ Reading stories2019-06-06T21:15:23Z Eyes: [LOG    ] {} (): navigating to storybook url: http://localhost:9000
⠹ Reading stories2019-06-06T21:15:23Z Eyes: [LOG    ] {} Page.<init>(): JSHandle:%cDownload the React DevTools for a better development experience: https://fb.me/react-devtools JSHandle:font-weight:bold
⠦ Reading stories2019-06-06T21:15:24Z Eyes: [LOG    ] {} Page.<init>(): JSHandle:ReferenceError: undefinedStoryName is not defined
    at Module.<anonymous> (http://localhost:9000/main.f790ddba604d943393ce.bundle.js:251:93)
    at Module../Sidebar/test/Sidebar.visuals.js (http://localhost:9000/main.f790ddba604d943393ce.bundle.js:264:30)
    at __webpack_require__ (http://localhost:9000/runtime~main.f790ddba604d943393ce.bundle.js:782:30)
    at fn (http://localhost:9000/runtime~main.f790ddba604d943393ce.bundle.js:150:20)
    at webpackContext (http://localhost:9000/main.f790ddba604d943393ce.bundle.js:216:9)
    at http://localhost:9000/main.f790ddba604d943393ce.bundle.js:29:12
    at Array.forEach (<anonymous>)
    at loadStories (http://localhost:9000/main.f790ddba604d943393ce.bundle.js:28:14)
    at ConfigApi._renderMain (http://localhost:9000/vendors~main.f790ddba604d943393ce.bundle.js:1497:20)
    at render (http://localhost:9000/vendors~main.f790ddba604d943393ce.bundle.js:1451:17)
2019-06-06T21:15:24Z Eyes: [LOG    ] {} Page.<init>(): JSHandle:`setOptions(options)` is deprecated. Please use the `withOptions(options)` decorator globally.
2019-06-06T21:15:24Z Eyes: [LOG    ] {} Page.<init>(): JSHandle:[HMR] connected
⠧ Reading stories2019-06-06T21:15:24Z Eyes: [LOG    ] {} (): Getting stories from storybook
2019-06-06T21:15:24Z Eyes: [LOG    ] {} Page.<init>(): JSHandle:getting stories from storybook. V2
2019-06-06T21:15:24Z Eyes: [LOG    ] {} (): Error: Evaluation failed: TypeError: Cannot read property 'children' of null
...

Impact:

Pretty serious. I had my CI running without the visual tests without any error

danielschwartz85 commented 5 years ago

Thanks for the issue, checking..