search

TheBrainFamily / chimpy

Develop acceptance tests & end-to-end tests with realtime feedback.
https://thebrainfamily.github.io/chimpy
MIT License
52 stars 28 forks source link

Headless testing with a real browser (not PhantomJS) #5

Open samhatoum opened 6 years ago

samhatoum commented 6 years ago

Issue by Sanjo Monday Apr 13, 2015 at 18:22 GMT Originally opened as https://github.com/xolvio/chimp/issues/6

See https://github.com/vvo/selenium-standalone#running-headlessly

samhatoum commented 6 years ago

Comment by samhatoum Wednesday Dec 16, 2015 at 08:34 GMT

do we need to do anything for this?

samhatoum commented 6 years ago

Comment by kamek-pf Monday Dec 21, 2015 at 18:24 GMT

Some instructions might be nice if this is already supported. Especially in a Meteor environment, it seems like the abstraction level is higher than in plain Node and we have less control over things.

@samhatoum I found a link you posted on #169 about xvfb. Looks like a good option (PhantomJS does have some issues), however, I can't figure out how to actually run the tests within the headless browser.

samhatoum commented 6 years ago

Comment by riiiiizzzzzohmmmmm Tuesday Dec 22, 2015 at 19:23 GMT

@Sanjo @samhatoum First, thank you both so much for all you've done to bring bonafide testing techniques to the Meteor community.

I began working wth Velocity, Cucumber, and Jasmine towards the end of the summer. I've got a nice little regression suite that I can run on my mac just fine. Now, I'm working on my CI setup and simultaneously trying to keep up with the deprecation of Velocity, et. al., and migration to chimp. I'm having some trouble, though.

I've got an Ubuntu 14.04 server. I've got Node 4.2.3, NPM 2.14.7, OpenJDK 1.8.0_72, Google Chrome 47 binary, ChromeDriver 2.20. I've also got meteor installed. When I boot my meteor app and tail the cucumber logs (old school Velocity style, I know), I get to the point where it says that the mirror is started and "Cucumber is running," but then it crashes saying that ChromeDriver timed out while attempting to start:

=> App running at: http://localhost:51913/
[velocity] Mirror has established connection with Velocity.
[xolvio:cucumber] Cucumber is running
[chimp][hooks] BeforeFeatures hook had error
{ [Error: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.19.346067 (6abd8652f8bc7a1d825962003ac88ec6a37a82f1),platform=Linux 3.13.0-68-generic x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.63 seconds
Build info: version: '2.47.1', revision: '411b314', time: '2015-07-30 03:03:16'
System info: host: 'my-ip-internal', ip: 'my-ip', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-68-generic', java.version: '1.8.0_72-internal'
Driver info: org.openqa.selenium.chrome.ChromeDriver]
  message: 'unknown error: Chrome failed to start: exited abnormally\n  (Driver info: chromedriver=2.19.346067 (6abd8652f8bc7a1d825962003ac88ec6a37a82f1),platform=Linux 3.13.0-68-generic x86_64) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 60.63 seconds\nBuild info: version: \'2.47.1\', revision: \'411b314\', time: \'2015-07-30 03:03:16\'\nSystem info: host: \'my-ip-internal\', ip: \'my-ip\', os.name: \'Linux\', os.arch: \'amd64\', os.version: \'3.13.0-68-generic\', java.version: \'1.8.0_72-internal\'\nDriver info: org.openqa.selenium.chrome.ChromeDriver',
  type: 'RuntimeError' }
Error: unknown error: Chrome failed to start: exited abnormally
  at Object.Future.wait (/<home_folder>/.meteor/packages/xolvio_cucumber/.0.19.4_1.10ydvsz++os.linux.x86_64+web.browser+web.cordova/npm/node_modules/chimp/node_modules/fibers/future.js:420:15)
  at Object.<anonymous> (/<home_folder>/.meteor/packages/xolvio_cucumber/.0.19.4_1.10ydvsz++os.linux.x86_64+web.browser+web.cordova/npm/node_modules/chimp/node_modules/xolvio-sync-webdriverio/node_modules/xolvio-fiber-utils/index.js:48:19)
  at Object.<anonymous> (/<home_folder>/.meteor/packages/xolvio_cucumber/.0.19.4_1.10ydvsz++os.linux.x86_64+web.browser+web.cordova/npm/node_modules/chimp/lib/session-manager.js:137:21)
  at initBrowser (/<home_folder>/.meteor/packages/xolvio_cucumber/.0.19.4_1.10ydvsz++os.linux.x86_64+web.browser+web.cordova/npm/node_modules/chimp/lib/cucumberjs/world-helper.js:149:15)
  at Object.worldHelper.setupBrowserAndDDP (/<home_folder>/.meteor/packages/xolvio_cucumber/.0.19.4_1.10ydvsz++os.linux.x86_64+web.browser+web.cordova/npm/node_modules/chimp/lib/cucumberjs/world-helper.js:191:7)
  at /<home_folder>/.meteor/packages/xolvio_cucumber/.0.19.4_1.10ydvsz++os.linux.x86_64+web.browser+web.cordova/npm/node_modules/chimp/lib/cucumberjs/hooks.js:8:17
  at /<home_folder>/.meteor/packages/xolvio_cucumber/.0.19.4_1.10ydvsz++os.linux.x86_64+web.browser+web.cordova/npm/node_modules/chimp/lib/cucumber-wrapper.js:114:43

It seems like this has something to do with trying to start chrome not in headless mode (i.e. with xvfb). So, then I tried to just run my cuke tests with chimp --ddp pointed at my running meteor app, but then my references to Meteor environment variables (i.e. process.env.ROOT_URL) throw execptions.

Am I missing something? How do I run my Cucumber tests against Chrome on an Ubuntu server?

samhatoum commented 6 years ago

Comment by Sanjo Tuesday Dec 22, 2015 at 20:15 GMT

@riiiiizzzzzohmmmmm With the latest chimp, you no longer need the xolvio:cucumber package. So, remove it. I think you use a version of xolvio:cucumber that had a bug with the environment variables. If you still want to use xolvio:cucumber, make sure you use the latest version.

samhatoum commented 6 years ago

Comment by samhatoum Tuesday Dec 22, 2015 at 21:19 GMT

@kamek-pf take a look here: http://tobyho.com/2015/01/09/headless-browser-testing-xvfb/

It looks like there's an pm package to control xvfb - could be an interesting add to Chimp... hmmm... https://www.npmjs.com/package/xvfb

samhatoum commented 6 years ago

Comment by samhatoum Tuesday Dec 22, 2015 at 21:19 GMT

also, CI servers like Circle and Travis have this built in

samhatoum commented 6 years ago

Comment by kamek-pf Tuesday Dec 22, 2015 at 21:24 GMT

@samhatoum Yeah I'm using CircleCI and I just noticed it does work out of the box, awesome :)

samhatoum commented 6 years ago

Comment by riiiiizzzzzohmmmmm Wednesday Dec 23, 2015 at 06:13 GMT

So, yeah, it turns out I just needed to use screen to start an Xvfb session before running my Velocity tests. I also needed to set a DISPLAY environment variable to reference my Xvfb session before my cucumber tests would work (by which I mean just booting my meteor app and letting the xolvio:cucumber package run my tests against a headless chrome in meteor mirror). (@kamek-pf not sure if this is the same issue you were having.)

@samhatoum It does look like it would be cool if Chimp could leverage that xvfb package so the aforementioned step would happen seamlessly/transparently/automagically. I'd even be happy to take a crack at it and submit a PR, if you'd like.

@Sanjo I will definitely be migrating away from the xolvio:cucumber package soon, but that will definitely require some refactoring of my tests (fixtures, etc.) which is a step I wasn't quite ready to do yet. I'm kind of a noob and figuring things out as I go along. I wanted to see if I could get the specs that I have as they are right now to run on an Ubuntu server the same way I can currently run them on my mac.

My ultimate goal is to be able to kick off concurrent builds of N instances (of Meteor apps) on ephemeral EC2 instances, so I can run different test suites and deploy.

samhatoum commented 6 years ago

Comment by samhatoum Wednesday Dec 30, 2015 at 13:52 GMT

@samhatoum It does look like it would be cool if Chimp could leverage that xvfb package so the aforementioned step would happen seamlessly/transparently/automagically. I'd even be happy to take a crack at it and submit a PR, if you'd like.

Would love that! With tests of course :)

but that will definitely require some refactoring of my tests (fixtures, etc.) which is a step I wasn't quite ready to do yet.

you can still use them, just move them to a package called fixtures and it will work without any other changes. Post back if you want help

My ultimate goal is to be able to kick off concurrent builds of N instances (of Meteor apps) on ephemeral EC2 instances, so I can run different test suites and deploy.

You should check out https://github.com/xolvio/whirlwind

samhatoum commented 6 years ago

Comment by SimonSimCity Thursday Jan 05, 2017 at 13:36 GMT

Got it working. See: https://github.com/SimonSimCity/bitbucket-meteor-headless-browsers

samhatoum commented 6 years ago

Comment by jimmyhoneycutt Thursday Apr 13, 2017 at 14:22 GMT

Will chimpjs work with headless chromium? Are there any instructions on how to set this up?

samhatoum commented 6 years ago

Comment by migerh Wednesday Jun 14, 2017 at 05:26 GMT

@jimmyhoneycutt I tweaked the chimp config of our project to get it running with chrome headless: https://github.com/4minitz/4minitz/blob/9096b5ed2503f0a24c6460c64e75f0a9da4a5c02/.meteor/chimp_config_headless.js. To run chimp with that config provide it as a parameter chimp .meteor/chimp_config_headless.js <other chimp parameters>.

In the args array headless and disable-gpu are to get chrome headless, window-size is for nice hd screenshots and no-sandbox is to get it running on Travis CI. The isHeadless: true is just for us to detect in our tests if it's a headless chrome.

Unfortunately, X or xvfb is apparently still required for browser.keys() to work.

samhatoum commented 6 years ago

Comment by fatso83 Monday Jul 24, 2017 at 22:59 GMT

Unfortunately, X or xvfb is apparently still required for browser.keys() to work.

As of July 22, this is no longer the case, @migerh. ChromeDriver 2.31 has been released and contains fix to this bug.

samhatoum commented 6 years ago

Comment by SimonSimCity Tuesday Jul 25, 2017 at 06:03 GMT

@jimmyhoneycutt and everyone who 👍'ed it ...

The repository I linked here uses Docker to set up a virtual machine to run the tests of your meteor-project. Here's a quick reference of how to read these files: https://www.digitalocean.com/community/tutorials/docker-explained-using-dockerfiles-to-automate-building-of-images#dockerfile-commands-instructions It's quite straight forward.

You need to install xvfb, the jre as described in https://github.com/SimonSimCity/bitbucket-meteor-headless-browsers/blob/master/Dockerfile#L5 and some browsers (f.e. Firefox 45.4.0 and Chrome). I fixed the Firefox version because of a requirement in Selenium (https://github.com/SeleniumHQ/selenium/blob/master/java/CHANGELOG#L33)

If you're done with that, you can simply run your xfvb like following:

Xvfb :99 -ac -screen 0 1280x720x16 -nolisten tcp &
xvfb=$!
export DISPLAY=:99

And you can run your command that opens f.e. chrome which will then be visible on this virtual screen. The variable xvfb contains the process-id so you can kill xvfb by running

kill $xvfb

Some sugar ...

If you want to see this virtual screen, you can install x11vnc to run an x11 with an open connection for VNC. Run these command instead of the previous snippet in order to be able to connect to x11vnc:

x11vnc -storepasswd 1234 ~/.vnc/passwd
x11vnc -create -forever &
xvfb=$!

And run f.e. chrome.

I hope this helps you getting started on it :wink:

samhatoum commented 6 years ago

Comment by samhatoum Wednesday Jul 26, 2017 at 23:10 GMT

thank you @SimonSimCity!