search

dwyl / learn-nightwatch

:last_quarter_moon_with_face: Learn how to use Nightwatch.js to easily & automatically test your web apps in *real* web browsers.
584 stars 214 forks source link

SPIKE Can we run the e2e tests on TravisCI (or other CI)? #8

Closed nelsonic closed 7 years ago

nelsonic commented 8 years ago

Given that we have the selenium start/shutdown and test run in a single command npm run e2e, we could try to run this on Travis...

nelsonic commented 8 years ago

Sadly this did not work looks like the Selenium server did not start: https://travis-ci.org/nelsonic/learn-nightwatch/builds/137542228#L185 we don't have an easy way of debugging it... But I'm going to try and install Java before running the tests... http://superuser.com/questions/993718/can-i-install-java-8-on-travis-ci-for-a-python-project

nelsonic commented 8 years ago

No luck ... https://travis-ci.org/nelsonic/learn-nightwatch/builds/137544553#L326 😢

nelsonic commented 8 years ago

Looks like Travis-CI is not going to be an option... see: https://github.com/travis-ci/travis-ci/issues/4090 (for now...)

nelsonic commented 8 years ago

Hold on, found this: http://stackoverflow.com/questions/18456611/is-it-possible-to-set-up-travis-to-run-tests-for-several-languages which leads to http://entulho.fiatjaf.alhur.es/guias/how-to-use-node-along-with-other-language-on-travis-ci/ which suggests that we set the language in .travis.yml to Java and then run the tests in a script ... e.g:

language: java

install:
  - . $HOME/.nvm/nvm.sh
  - nvm install stable
  - nvm use stable
  - npm install

script:
  - npm test

This example is for Go but still relevant: https://github.com/OrangeTux/Goppetto/blob/develop/.travis.yml

nelsonic commented 8 years ago

Setting the language to Java did not work ... https://travis-ci.org/nelsonic/learn-nightwatch#L417

attempting: https://docs.travis-ci.com/user/customizing-the-build/#Installing-a-Second-Programming-language linked to by: http://superuser.com/questions/993718/can-i-install-java-8-on-travis-ci-for-a-python-project#comment1405598_993718

nelsonic commented 8 years ago

Note: This works on CodeShip without any issues: learn-nightwatch-codeship-passes learn-nightwatch-codeship-2 Codeship passes while Travis Fails... codeship-passes-travis-fails https://travis-ci.org/nelsonic/learn-nightwatch/builds/137568733#L138

nelsonic commented 8 years ago

Travis is pure frustration ... 😞 https://travis-ci.org/nelsonic/learn-nightwatch/builds/137568733#L138

nelsonic commented 8 years ago

Attempt to run the selenium jar file as a background process: http://stackoverflow.com/questions/12102270/run-java-jar-file-on-a-server-as-background-process

nelsonic commented 8 years ago

Attempt to use https://github.com/angular/webdriver-manager to run selenium...

nelsonic commented 8 years ago

LoL: https://travis-ci.org/nelsonic/learn-nightwatch/builds/137772914#L230

nelsonic commented 8 years ago

Check status of selenium server before running the tests: http://localhost:4444/wd/hub/status selenium-status-output

from: http://stackoverflow.com/questions/6517501/selenium-2-how-to-check-that-server-is-running-and-stop-the-server

nelsonic commented 8 years ago

Selenium is Running! https://travis-ci.org/nelsonic/learn-nightwatch/builds/137849262 travis-loads-selenium

But I cannot access it! travis-cannot-connect-to-selenium

nelsonic commented 8 years ago

When we attempt to start selenium in nightwatch.json we get: https://travis-ci.org/nelsonic/learn-nightwatch/builds/137876826#L192 travis-fails-when-nightwatch json-set-to-start-selenium

nelsonic commented 8 years ago

Trying the advice in: https://github.com/travis-ci/travis-ci/issues/2843 to sleep ... before running the tests!

nelsonic commented 8 years ago

https://travis-ci.org/nelsonic/learn-nightwatch/builds/137880716#L245 Ok, I'm shelving this for now. Its taken way too much time to do something that should be simple...!! These guys appear to have it working: https://github.com/Khan/KaTeX/blob/master/.travis.yml which lead me to: https://github.com/SeleniumHQ/docker-selenium

But I don't have time to investigate this further today... if anyone else has time to try and debug this, please go for it! $100 bounty for successfully running Selenium (Nightwatch) tests on Travis-CI. 💲

nelsonic commented 8 years ago

Sadly, Travis-CI is still not cooperating: https://travis-ci.org/nelsonic/learn-nightwatch/builds/140380579#L205

Appears we nee to use Xvfb see: http://stackoverflow.com/a/23673054/1148249

See instructions on Travis-CI: https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-a-GUI

nelsonic commented 8 years ago

Adding Xvfb as instructed by Travis-CI did not work. https://travis-ci.org/nelsonic/learn-nightwatch/builds/140396366

travis-learn-nightwatch-fail
nelsonic commented 8 years ago

I really don't have time to mess about with Travis-CI anymore ... if anyone wants to pitch in and figure this out I have £100 with your name on it. (preferably with the ability to take/save screenshots...)

This was the state before I gave up: my .travis.yml file:

language: node_js
node_js:
  - 4.4.6
before_install:
  - java -version
before_script:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
  - sleep 3 # give xvfb some time to start

Outcome:

learn-nightwatch-travis-ci-fail-1of2 learn-nightwatch-travis-ci-fail-2of2

@des-des / @jrans you gents appear to have some practice/experience of using Nightwatch in cmt-integration ... if you have time to look at this, just invoice me for how ever long it takes you to get it working! 😉

amilajack commented 8 years ago

@nelsonic would a PR for this still be welcome? Think I might give it a go.

amilajack commented 8 years ago

I'm getting the error:

TypeError: Path must be a string. Received { url: 'http://selenium-release.storage.googleapis.com/2.53/selenium-server-standalone-2.53.0.jar'

Seems like env2 isn't configured properly in the config.

nelsonic commented 8 years ago

@amilajack I think the error relates to the latest version of download not working with node v6 see: https://github.com/dwyl/learn-nightwatch/issues/24 we first need make selenium-download compatible with node v6.x ... I forked the repo: https://github.com/dwyl/selenium-download and started to debug why it wasn't working ... (you are a contributor if you want to take a look)

nelsonic commented 8 years ago

@Danwhy & @SimonLab what are you doing differently to run your Nightwatch tests on Travis-CI ? They appear to run without issues: https://travis-ci.org/TheScienceMuseum/collectionsonline/builds/169499325#L1957 running this command: https://github.com/TheScienceMuseum/collectionsonline/blob/master/package.json#L71 CC: @Conorc1000

Danwhy commented 8 years ago

It could be because we're running ours through saucelabs

nelsonic commented 8 years ago

@Danwhy next time we're in the same room please walk me through it so I understand. thanks! 😉

justinmc commented 8 years ago

For whatever it's worth, I seem to have gotten my project to work with Nightwatch tests on Travis using this guide, but I had to settle with only using phantomjs: https://github.com/Autodesk/molecule-2d-for-react/pull/15

nelsonic commented 8 years ago

@justinmc nice work! thanks for confirming you got it working with PhantomJS! 👍

nelsonic commented 8 years ago

@jackcarlisle this is how I log progress in a "Spike"... it lets someone else see what has been attempted and can pick up the baton...

jackcarlisle commented 8 years ago

@nelsonic 👍 I'll do the same for the karma spike

gregorybesson commented 7 years ago

Hello guys,

I've been able to make it work with Chrome in Travis: https://github.com/abecms/abecms/blob/master/.travis.yml

  1. You have to use the beta build from travis

    sudo: required
dist: trusty

  2. You launch xvfb + install the last stable version of Chrome

    before_install:
- export CHROME_BIN=/usr/bin/google-chrome
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sudo apt-get update
- sudo apt-get install -y libappindicator1 fonts-liberation
- wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
- sudo dpkg -i google-chrome*.deb

  3. last but not least, switch to the last version of jdk:

    script:
- jdk_switcher use oraclejdk8

The nightwatch config file is here:https://github.com/AdFabConnect/abejs/blob/master/nightwatch.conf.js

Hope it will work for you ;)

Conorc1000 commented 7 years ago

Since this issue was made this short piece has been added to the readme. "Running your Nightwatch tests on CircleCi."

qacwnfq commented 7 years ago

@gregorybesson Thanks, it works! For me the deciding part seems to have been export DISPLAY and starting xvfb.

gregorybesson commented 7 years ago

@qacwnfq U welcome. Indeed, we must start xvfb for chrome to be able to launch

ritz078 commented 7 years ago

@nelsonic Tests running on Travis with saucelabs

screen shot 2017-03-15 at 1 27 23 am

Here is the full code https://github.com/ritz078/embed.js/pull/228

nelsonic commented 7 years ago

@ritz078 nice one. 👍 If you have time to send us a Pull Request explaining the steps to get it setup using sauce connect, it would be massively appreciated! ❤️ (once PR is merged we can send you PayPal or Amazon credit 😉 )

ritz078 commented 7 years ago

I will create a PR.

nelsonic commented 7 years ago

@ritz078 thank you for your fantastic contribution! 👍 I have reviewed & merged your Pull Request. ✅ Please email me to claim your reward! 🥇

kalote commented 6 years ago

I just read this full issue that looked like a story ... and that was a super funny one!

Nice job!