garris / BackstopJS

Catch CSS curve balls.
http://backstopjs.org
MIT License
6.71k stars 602 forks source link

Backstopjs hangs after 'reference' command #799

Open rickdroce opened 6 years ago

rickdroce commented 6 years ago

Once in a while, I experience a bug where backstopjs hangs in the terminal after the "reference" command is executed, without having the option to run further commands.

This wasn't a problem when I used to run tests manually, but now that I run backstopjs in a Jenkins job, it's dangerous since the job just builds forever since it doesn't receive any error.

screenshot from 2018-06-13 10-18-23

As you can see in the screenshot the command executed successfully and then hang there for a few hours until my manual abortion of the build.

garris commented 6 years ago

Is it possible to migrate to the puppeteer engine? I highly recommend it if you can. The implantation is more stable.

If you can’t migrate you could try adding more memory to your container.

rickdroce commented 6 years ago

Right now I already set a --shm-size=2gb so I don't think it would need more.

I was considering the migration to CasperJs because I need to login to the website in order to make a couple of tests. By the way, for now, I'm not really satisfied with the results, the time needed for the full test went to 10mins from 4 mins And there are actually some layouts that break down

Gonna make some more tries

Kenith commented 6 years ago

Which engine you used for test?

If phantomjs, yes, the phantomjs has the hang issue... dam...

Solve the hang issue by following:

  1. Set the following settings in backstop.json
"engine": "phantomjs",
  "casperFlags": [
    "--ignore-ssl-errors=true", 
    "--ssl-protocol=any",
    "--engine=phantomjs"
  ], 

2.Add following settings in casper/onBefore.js to avoid the PhantomJS hang issue. If not set this, the phantomjs has high possibilities to hang there forever.

casper.page.settings.resourceTimeout = 300000;
  casper.options.waitTimeout = 300000;
  casper.page.onResourceTimeout = function() {
    console.log("Phantomjs failed to donwload the resource in 5 minutes..."); 
    casper.bypass(1);
  };

If other engines, you could get the stable expertise here: https://github.com/garris/BackstopJS/issues/761#issuecomment-390492390

For docker use, you may add --privileged -v /dev/shm:/dev/shm to docker run

rickdroce commented 6 years ago

I took a look at the issue you linked and the only flag I was missing was "--hide-scrollbars" I'm using chromy I also added the "--privileged" arg to my docker run

I'll make some tests.

N.B. I also noticed that the 'hang' happens the most when the job is triggered by another build. So maybe it could really be a memory issue. Here are my args for the container: --shm-size=2gb --cap-add=SYS_ADMIN --privileged