search

open-lms-open-source / moodle-plugin-ci

Assist with running a Moodle plugin in Travis CI
https://blackboard-open-source.github.io/moodle-plugin-ci/
GNU General Public License v3.0
42 stars 37 forks source link

With Travis - Behat fails, selenium not running. #70

Open roperto opened 6 years ago

roperto commented 6 years ago

It seems like a Java problem (surprise surprise)

Workaround: add sudo: required to .travis.yml

stronk7 commented 6 years ago

As commented @ 4f97e21, I've tried with sudo: required and it helped with behat executions but, at the same time, introduced a really annoying behavior with random failures all over the place. See:

So, particularly, I'm back to sudo: false + disabled behat (it's not critical in my plugin). And at least that way I get more consistent results. See:

Just sharing, not really sure which the selenium / java problem is. Annoying. Ciao :-)

roperto commented 6 years ago

Hi @stronk7

I am also getting inconsistent problems updating packages and such.

I disable while developing, but when merging I still want the peace of mind of behat tests passing

Please let us know if you find a workaround, Eloy ...

Thanks

mackensen commented 6 years ago

I'm seeing the same issues with my plugins. Behat's important to me, so I'm setting sudo: true and re-running failures. This is with version 2 of the plugin.

polothy commented 6 years ago

Do you guys have more example of build failures? Eloy's failure has nothing to do with Behat, it's failing to install Java8 (oracle-java8-installer, oracle-java8-set-default) because for whatever reason, some of the builds didn't find oracle-java8-set-default package, while others did. This is only needed for the mustache linting IIRC. You could try Java9, but betting that wont work.

roperto commented 6 years ago

hi @mrmark .. I have more examples but that's exactly the same problem.

We currently have a custom docker image to run moodle-plugin-ci to be used in our GitLab, but it has a lot of custom code.

If we make it more flexible do you think it would be a approach to running it Travis? Hopefully would speedup part of initialisation (it would only pull changes in Moodle since the image was built, composer init, npms, etc) and less prone to Travis infrastructure changes. The initialisation process would still run as phpunit/behat init is required after the plugin is added.

If I went ahead, would you be interested in that or do you think its a dead end?

mackensen commented 6 years ago

@mrmark Sure. This is for the https://github.com/LafColITS/moodle-tool_hidecourses plugin:

scara commented 6 years ago

@mackensen @mrmark here is my working example w/ sudo: false: https://travis-ci.org/scara/moodle-local_twittercard/builds/330854428 .

HTH, Matteo

stronk7 commented 6 years ago

Hi,

just note that only @javascript scenarios require selenium + real browser to be working, so it's normal, for people not having such a tag to get behat passing.

In fact it was working ok for my plugin, but because of a little restriction with checkboxes I needed to add the @javascript tag to one of my features.. and there is where the problem started to happen consistently.

Just to explain/clarify why tests seem to work to some people without problem. It's not magic or luck. Only @javascript scenarios are affected by selenium not working

stronk7 commented 6 years ago

OT: just came to my brain... could it be some problem with IPv4/IPv6 ? I've had a flash that some time ago I had to force IPv4 in some places (-Djava.net.preferIPv4Stack=true) because not all the stack was able to work with IPv6 properly (and the system insisted to use IPv6 by default).

As said... just a blind shot!

polothy commented 6 years ago

If we make it more flexible do you think it would be a approach to running it Travis? Hopefully would speedup part of initialisation (it would only pull changes in Moodle since the image was built, composer init, npms, etc) and less prone to Travis infrastructure changes. The initialisation process would still run as phpunit/behat init is required after the plugin is added.

That was proposed in #57 to use moodle-docker, but I was worried about performance. I'd likely only want to move to a core supported/maintained container if we ever went in that direction. I do agree though, there have been a lot of build breaks due to Travis updates, which is too bad.

Thanks everyone for all the examples, feedback!

scara commented 6 years ago

It's not magic or luck. Only @javascript scenarios are affected by selenium not working

TNX @stronk7! My bad for not thinking at it: shame on Matteo 😊 !

polothy commented 6 years ago

Having lots of troubling figuring out the problem. Have a build I'm debugging here: https://travis-ci.org/moodlerooms/moodle-plugin-ci/jobs/334948327

Anyone know what's going on? Any guesses?

roperto commented 6 years ago

Hi @mrmark , as you said PHP server (behat site) looks fine.

I'd investigate towards the selenium server, I am guessing a driver/selenium problem from "UnsupportedCommandException" at https://travis-ci.org/moodlerooms/moodle-plugin-ci/jobs/334948327#L1278 -- maybe a mismatch of firefox version and selenium driver?

scara commented 6 years ago

@mrmark @roperto Bad request looks like coming from a 400 HTTP Status: what about https://github.com/seleniumhq/selenium/issues/2074 ?

HTH, Matteo

polothy commented 6 years ago

Thanks @scara, that might be the problem, I commented in https://github.com/travis-ci/travis-ci/issues/8711

There is cat /etc/hosts with sudo false/true:

If it is indeed a problem with /etc/hosts then there is nothing we can do about it because in order to fix it, we would need sudo, so might as well just use sudo: true in your build file until Travis fixes the container.

scara commented 6 years ago

Great @mrmark! At the end @stronk7 was right about IPv4: we need to force the binding to IPv4 since IPv6 support is broken which means -Djava.net.preferIPv4Stack=true on the Java command line and php -S 127.0.0.1:8080 to force the builtin-in server bind to IPv4 i.e. no localhost here. You could try first w/ PHP and then if it won't work force IPv4 at Java side too.

For the record see another example using sudo: required - which is slower among the other cons -, read carefully the commit message: https://github.com/mrkn/ruby/commit/ebffe7811860b81eee260beeeef6d658a49eb54c.

HTH, Matteo

polothy commented 6 years ago

Thanks for the continued feedback, but doesn't appear to work unfortunately, https://travis-ci.org/moodlerooms/moodle-plugin-ci/builds/335718087

scara commented 6 years ago

Hi @mrmark, to recap, the failure is now related to the new settings towards a "pure IPv4 net env":

~~Before leaving this path, what about give -Djava.net.preferIPv4Addresses=true also a try? That could exclude any IPv6 attempt since from your logs it's pretty clear that the (first) failure happens in opening a session against Selenium~~.

After more digging the web I found https://github.com/seleniumhq/selenium-google-code-issue-archive/issues/3280 and https://github.com/SeleniumHQ/selenium/commit/1ada6dd7056c0dc3e8f6c96fa896acb782d52b04: it looks like there's no need to force IPv4 but IIUC to force the hosts preference to localhost. Guessing that Bad Request here is coming from a mismatching between the Host HTTP header coming with the request and the info taken by the Firefox built-in web server. If that is true, I'm not sure how to test it since we need to inject a value for the (new) pref webdriver_firefox_allowed_hosts.

HTH, Matteo

scara commented 6 years ago

FYI yesterday I've also filed https://github.com/travis-ci/travis-ci/issues/9167 which could solve our issue here if the Bad Request is actually due to the archived issue I've found.

At the moment I'm not sure if we'd need some IPv4 tuning after that fix: will see.

polothy commented 6 years ago

Thanks @scara for your help, I think I'll wait for the Travis issue to get some traction before I try anything else on this.

polothy commented 6 years ago

With https://github.com/travis-ci/travis-ci/issues/9167 fixed, still doesn't work :(

Re-ran my master branch with no luck and same for the debugging one. /etc/hosts now looks like this:

::1  ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.4  travis-job-moodlerooms-moodle-plugin-335718089.travisci.net travis-job-moodlerooms-moodle-plugin-335718089
127.0.0.1 localhost localhost
scara commented 6 years ago

Hi @mrmark, I'm out of smart ideas than trying to revert https://github.com/moodlerooms/moodle-plugin-ci/commit/fe9db9abb0c848998c388cfa7173b16939649b6f#diff-0 to let Selenium be accessed via localhost and not via 127.0.0.1, since we found that this didn't helped in the attempt of forcing IPv4 and maybe this revert together with the recent change in /etc/hosts could help the resolution of the host in the Firefox built-in web server.

polothy commented 6 years ago

Minor update:

First, as of 2.3.0, recommending sudo: true for builds that run Behat, which I think most of you are already doing.

Second, think I sort of verified that the problem in the container build system is with /etc/hosts, in particular, this line: ::1 ip6-localhost ip6-loopback. My funky reproduction was to do this:

  1. I switched my Travis file to use sudo: true, build passes.
  2. I edited the /etc/hosts to add ::1 ip6-localhost ip6-loopback, build fails.
  3. I edited the /etc/hosts to add ::1 localhost ip6-localhost ip6-loopback, build passes.

From what I can tell, ::1 localhost was removed in https://github.com/travis-ci/travis-ci/issues/8780. Unsure if we will be able to get it back.

mudrd8mz commented 6 years ago

Thanks @mrmark for the update. I am affected by this too (e.g. https://travis-ci.org/mudrd8mz/moodle-block_course_contents/jobs/336000695).