search

garris / BackstopJS

Catch CSS curve balls.
http://backstopjs.org
MIT License
6.76k stars 603 forks source link

WaitTimeoutError during test #537

Open ryuran opened 7 years ago

ryuran commented 7 years ago

Helo!

After an error on Travis (https://github.com/OnetapInc/chromy/issues/62), I try to reduce async limit to pass test.

asyncCaptureLimit: 5,
asyncCompareLimit: 50

But nothing change 😢 : https://travis-ci.org/20minutes/colette/builds/271137302

Then I try with

asyncCaptureLimit: 1,
asyncCompareLimit: 1

https://travis-ci.org/20minutes/colette/builds/271199288 I don’t know why it’s green even if i have no result of the test…

garris commented 7 years ago

Did you try to reduce to 1?

ryuran commented 7 years ago

@garris yes, my second try is at 1:1 It’s better but travis return positive whereas backstop was not able to finish test and I have some errors in logs.

ChromyJS returned an unexpected error while attempting to capture a selector. { WaitTimeoutError: evaluate() timeout
    at EvaluateTimeoutError.ExtendableBuiltin (/home/travis/build/20minutes/colette/node_modules/chromy/dist/error.js:44:28)
    at EvaluateTimeoutError.TimeoutError (/home/travis/build/20minutes/colette/node_modules/chromy/dist/error.js:73:128)
    at new EvaluateTimeoutError (/home/travis/build/20minutes/colette/node_modules/chromy/dist/error.js:118:145)
    at Chromy._evaluateWithReplaces$ (/home/travis/build/20minutes/colette/node_modules/chromy/dist/document.js:484:21)
    at tryCatch (/home/travis/build/20minutes/colette/node_modules/babel-runtime/node_modules/regenerator-runtime/runtime.js:65:40)
    at Generator.invoke [as _invoke] (/home/travis/build/20minutes/colette/node_modules/babel-runtime/node_modules/regenerator-runtime/runtime.js:299:22)
    at Generator.prototype.(anonymous function) [as throw] (/home/travis/build/20minutes/colette/node_modules/babel-runtime/node_modules/regenerator-runtime/runtime.js:117:21)
    at tryCatch (/home/travis/build/20minutes/colette/node_modules/babel-runtime/node_modules/regenerator-runtime/runtime.js:65:40)
    at invoke (/home/travis/build/20minutes/colette/node_modules/babel-runtime/node_modules/regenerator-runtime/runtime.js:155:20)
    at /home/travis/build/20minutes/colette/node_modules/babel-runtime/node_modules/regenerator-runtime/runtime.js:167:13
    at <anonymous> name: 'WaitTimeoutError' }
garris commented 7 years ago

I have just verified that asyncCaptureLimit and asyncCompareLimit are working as designed. At this point I think you need to look at why your particular test is failing.

ryuran commented 7 years ago

I add log for test scenario label. It seems fail on ./docs/item-theme-skins.html or the next one ./docs/item-theme-styles.html. I check this files with html validator, they are valid. So I’m blocked. All is fine in local environment.

kiran-redhat commented 7 years ago

@ryuran 1. are you about to find that in chrome console ?

  1. Do you see sale error with asyncCaptureLimit: 1,?

If 1 &2 are yes, clone latest master and test. asyncCaptureLimit and asyncCompareLimit we're working fine for me

ryuran commented 7 years ago

@kiran-redhat

  1. I don’t know how to return more info from browser in travis-CI logs
  2. This travis build is running with asyncCaptureLimit: 1 https://travis-ci.org/20minutes/colette/builds/271948792 I have the error
kiran-redhat commented 7 years ago

for number 1: open your app in Chrome browser press f12 then click on console then copy and paste your selector and hit enter. if you find element

wait for your selector to be visible (probably chromy.visible() )

On Sep 5, 2017 9:24 PM, "Yvain Liechti" notifications@github.com wrote:

@kiran-redhat https://github.com/kiran-redhat

  1. I don’t know how to return more info from browser from travis-CI
  2. This travis build is running with asyncCaptureLimit: 1 https://travis-ci.org/20minutes/colette/builds/271948792 https://travis-ci.org/20minutes/colette/builds/271948792 I have the error

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/garris/BackstopJS/issues/537#issuecomment-327121456, or mute the thread https://github.com/notifications/unsubscribe-auth/AQUrIz_uD5ROOyWVDt21w2YFEBIEIiIwks5sfRNCgaJpZM4PK4Y4 .

ryuran commented 7 years ago

No error in my local chrome. Backstop.js run fine in local environment with all test to green.

I update backstopjs:

└─┬ backstopjs@3.0.24
  └── chromy@0.5.2

And I try to change delay to 200 (it was 100), but it stops always on the same test. Without error this time but with travis timeout: https://travis-ci.org/20minutes/colette/builds/271982035

ryuran commented 7 years ago

It seems fail most often on test with <img>

reynaldoapb commented 7 years ago

Try to kill chrome with shell command taskkill /F /IM chrome.exe and run everything again.

axe312ger commented 7 years ago

Turning it down to 

asyncCaptureLimit: 1,
asyncCompareLimit: 1

worked for me on travis :)

mhavelant commented 7 years ago

Hi!

I got the same error. The async limits at 1 did not help, however setting "selectors" to "viewport" from "document" did. So it might be something with chromy with 'document' as the selector.

Edit: Setting asyncCaptureLimit to 1 AND not using "document" together worked. Setting the capture limit to something higher broke it.

garris commented 7 years ago

What version of chrome and backstop are you using?

mhavelant commented 7 years ago

The newest ones:

Note: I'm running under a dockerized Ubuntu 16.04

tfrijsewijk commented 7 years ago

I'm experiencing the same issue, with the same versions as mhavelent

garris commented 7 years ago

Hello all. Thank you very much for posting.

It may be possible that #561 (running out of memory) is related to this issue — which only seems to happen on Linux.

@vdekov posted a fix for #561 here... https://github.com/garris/BackstopJS/pull/571

Would it be possible for someone to pull this PR and see if the problem resolves?

mhavelant commented 7 years ago

For me, the error

ChromyJS returned an unexpected error while attempting to capture a selector. { WaitTimeoutError: evaluate() timeout...

is thrown before the 'BackstopTools have been installed.' step in chromy. So long before comparisons are even considered, in the reference command.

So the PR, although sounding insanely good overall, I don't think will help here.

At least for me, it seems that the 'document' selector is not working with Chromy.

garris commented 7 years ago

Oh. Crap. You’re right. Ok — scratch that. 😕

kevineye commented 7 years ago

I was getting this WaitTimeoutError, too, not running natively, but when switching to run in docker. Happened more often with higher asyncCaptureLimit and more complex pages, but even sometimes with asyncCaptureLimit: 1.

Eventually tracked the problem down to chrome dying with a bus error, and finally tracked this down to running out of space in /dev/shm. Docker allocates 64MB to /dev/shm by default.

Running docker with --shm-size 512m fixed the issue for me. Lets me run with default asyncCaptureLimit: 10.

garris commented 7 years ago

@kevineye Nice debugging — thanks for posting!

garris commented 6 years ago

cross referencing this issue... https://github.com/garris/BackstopJS/issues/603#issuecomment-346478523