[percy] TypeError: Cannot read properties of undefined (reading 'message')

alextreppass commented 1 year ago

The problem

Percy intermittently fails for us with TypeError: Cannot read properties of undefined

Environment

Node version: 16.4.2
@percy/cli version: 1.25.0
Version of Percy SDK you’re using (@percy/storybook): 4.3.6
OS version: Linux (github actions CI runner). Happens very occasionally on OSX locally too
Type of shell command-line [interface]: bash

Details

Sometimes, the Percy runner blows up and doesn't take any snapshots. This appears to be due to the 'force stop' code path in the percy cli, from a catch block

> [percy] Stopping percy...
> [percy] Build not created
> [percy] TypeError: Cannot read properties of undefined (reading 'message')

The type error appears to be due to this line of code in src/utils.js: https://github.com/percy/percy-storybook/blob/bb665fd94c2261e88768d265df66a7742f6006d1/src/utils.js#L128-L130

^ that line assumes that error is a defined object, and so causes a TypeError at runtime if error isn't an object (e.g is undefined)

Debug logs

https://gist.github.com/alextreppass/b03b90b90f5447c1f43acba01b2c2100 - these are the --verbose logs, unfortunately, Percy isn't giving us much here

Code to reproduce issue

Cannot reliably reproduce, unfortunately. This happens a few times a day for us on pull request branch CI jobs (Linux)

I have had it happen a small # of times running the command locally (OSX).

What has been helpful is a 'while' loop to re-run percy locally until it fails:

percy-report.sh

yarn percy storybook ./storybook-static --debug
status=$?
[ $status -eq 0 ] && (echo "Percy was successful"; exit 0) || (echo "Percy failed"; exit 1);

command to run until it fails: (be very careful you leave --debug in or it'll eat your quota)

while yarn percy:report; do :; done

alextreppass commented 1 year ago

Update: after patching out the assumption that error is defined, and upon further debugging, it seems the error is thrown from the snapshots.js catch block here. The error is undefined at runtime:

https://github.com/percy/percy-storybook/blob/bb665fd94c2261e88768d265df66a7742f6006d1/src/snapshots.js#L226-L229

Unfortunately, the try block above is doing a lot of work, so it'll be hard to pin-point. I will attempt to add more debug logging

alextreppass commented 1 year ago

Ok. New update: it's failing inside this call here: https://github.com/percy/percy-storybook/blob/bb665fd94c2261e88768d265df66a7742f6006d1/src/snapshots.js#L205

It never gets to the following yield page.snapshot(options); instruction.

It appears the evalSetCurrentStory execution in the context of the current page is not successful.

Here's a log of exceptionDetails inside your page.eval function:

{
  exceptionId: 1,
  text: 'Uncaught (in promise)',
  lineNumber: 0,
  columnNumber: 0,
  exception: { type: 'undefined' }
}

... that's as far as I can take it I think.

I cannot see any errors when running + visiting the built storybook in storybook-static manually.
The storybook itself is loading ok, because I can see Percy is running through stories and skipping a number of them which we are not visregging.
For some reason, the first attempt to snapshot fails sometimes, seemingly with an uncaught promise rejection. It is not specific to the story: if I opt-out of visreg for that story, another one will start failing — whichever the first one needing a snapshot happens to be.

I am wondering if it is the 5000ms Storybook object not found on the window promise rejection, where it is looking for the storybook channel to emit config to? ...If so, the error message is not making its way back to the runner. That might be a red herring though.

itsjwala commented 1 year ago

Thanks for reporting, this is similar to #715 and there are some suggestions mentioned there.

percy / percy-storybook