Setup client-side test infrastructure

[ebruchez]

Some tests are painful to do manually, in particular:

• Data Capture Permissions
• Autosave and Permissions Test
• Form Builder Permissions

These tests are also hard to automate with Selenium due to the need to login/logout multiple times.

We should look into hooking up PhantomJS. It supports setting custom headers which would easily allow testing with multiple users using our header-based auth.

[ebruchez]

So PhantomJS has a third-party Selenium driver called https://github.com/detro/ghostdriver, but it seems to have maintenance issue, in particular with PhantomJS 2.0.

That project now also links to a newer driver, https://github.com/MachinePublishers/jBrowserDriver. So that's something to look at.

[ebruchez]

It seems that jBrowserDriver is not a PhantomJS driver but its own thing. Not sure if that makes a difference.

[ebruchez]

Interesting. jBrowserDriver uses WebKit included with Java 8 (part of JavaFX).

[ebruchez]

It seems that jsdom might be the current hotness.

This would require running the tests from Node.js. Also, jsdom "does not perform layout or rendering, and it does not support navigation between pages". But this has benefits too: speed, no dependencies other than Node.js to install, and that still allows testing a lot of what we need to test.

[ebruchez]

If we use jsdom or similar, tests will run within a JavaScript environment. It is not likely that we can use the JVM for that (even though there are talks here and here about using Nashorn but it's unclear how successful that is, and the Avatar.js project appears without activity for about 2 years - sad).

If we don't use the JVM, we will use Node.js to run the client-side tests, which means there will be a need to communicate between Node and the JVM. Launching Tomcat as we do now from ant is verbose and brittle.

So we should look at the following:

1. Run tests with Node.js by using the built-in Scala.js support for tests.
2. Launch an embedded version of Tomcat, using the Tomcat embedding API.
3. Communicate between the two with HTTP, as is normally the case between client and server.

[ebruchez]

More confirmation that Avatar.js is dead here and here.

[ebruchez]

Did some exploratory work with formBuilderClient in build.sbt:

• jsEnv := JSDOMNodeJSEnv().value: enable jsdom
• enablePlugins(ScalaJSJUnitPlugin): enable JUnit
• remove dependency on runtimeDOM
• jsDependencies += "org.webjars" % "jquery" % "1.12.0" / "1.12.0/jquery.js": add jQuery dependency
• npm install jsdom
• persistLauncher in Test := true otherwise getting an error
• also need an Scala.js main class otherwise launcher is not produced and we get an error

[ebruchez]

With the setup above, we can run JUnit tests written in Scala which run with Node.js and jsdom.

This is still very experimental. As a next step, we need to:

• [x] see how we can run this on travis (update .travis.yml to npm install stuff)
• [ ] move to be an integration test (it)
  • 2017-08-17: had issues getting it:test to work with Scala.js
• [x] run dom tests with sbt (JVM only for now, later JS too)
• [ ] run remaining tests with sbt using JUnit (JVM only)
• [x] start/stop embedded servlet container
• [x] consider using Node's child_process to run Docker commands
  • this would alleviate issues with Undertow.stop()
• [ ] run at least one test connecting to servlet container
• [ ] write/move useful tests

[ebruchez]

Starting/stopping a server and running tests against them is what's covered in sbt by integration testing.

The Tests.Setup() and Tests.Cleanup() options are called by sbt before and after running tests, so maybe that's where we need to hookup starting our embedded server.

[ebruchez]

We don't have to use Tomcat. We could use Jetty or even Undertow.

[ebruchez]

Note that I didn't manage yet to get JUnit working with ScalaTest assertions as we do so far for our JVM tests written in Scala.

[ebruchez]

Made ScalaUtilsTest run by sbt for both JVM and JS. Uses ScalaTest directly without JUnit.

[ebruchez]

We already use Docker for database tests to make Oracle and other databases available to tests.

Instead of embedding Undertow (or other), an alternative would be to use a Tomcat Docker image. It is fairly easy to mount host directories and files into the Docker container.

Would this be better of worse?

Could this be worse for the build time because downloading the Docker image might be slower than getting the Undertow dependency via sbt (which can be cached)? Maybe, but that is not certain.

Using Undertow directly via a Java API is probably more flexible and doesn't require configuration files.

[ebruchez]

One issue is how to run setup and cleanup.

sbt has Tests.Setup and Tests.Cleanup. Now we can hookup to that (verified). However the Scala code that runs within sbt does not have access to test code compiled with sbt. We would have to place it under project, I think, which is annoying because that code cannot leverage lots of utilities we need.

We could use the ClassLoader option of Tests.Setup(). This gives access to the ClassLoaderof the compiled test classes, and then with reflection we can run some setup/cleanup methods. But this only works if the tests do not run in forked mode, which we use.

[ebruchez]

Maybe for now we just do this initialization within a ScalaTest test suite?

[ebruchez]

Ah, but how will this work if we need to run Scala.js tests?

[ebruchez]

Good progress on this for #1529:

• can start/stop Docker instances from Scala.js
• can start a Tomcat 8 container pointing to our orbeon-war with custom configuration files
• can load a simple XForms and it initializes within JSDOM

[ebruchez]

Following discussion with @avernet, it might be simplier to just run external processes directly from Node rather than using the Undertow remote control. That would make things simpler and avoid the hang I am getting with stopping that server at times.

See child_process.

[ebruchez]

Now one issue is that when we launch commands, we need to know the project's base directory. We need to check whether we obtain the right thing with Node. With the current sbt-based start/stop, we simply pass the values we get from sbt, so we are sure to have the right ones.

[ebruchez]

If we find a way, we could also pass parameters to the test from sbt. Could ConfigMap help?

[ebruchez]

Cannot find a way with ConfigMap so far. So working around this with BuildInfo. This way we can still pass values obtained from sbt to the test.

Using Node to deal with docker works just fine. So we can remove our Undertow server.

One issue is that ScalaTest hangs after completing tests, with this:

> last orbeonWarJS/test:testOnly
[debug] Running TaskDef(org.orbeon.fr.OrbeonClientTest, org.scalajs.testadapter.FingerprintSerializers$DeserializedSubclassFingerprint@7f40b38, false, [SuiteSelector])
java.util.concurrent.TimeoutException: Futures timed out after [59988191119 nanoseconds]
    at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
    at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
    at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
    at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
    at scala.concurrent.Await$.result(package.scala:107)
    at org.scalajs.jsenv.AsyncJSRunner$class.await(AsyncJSRunner.scala:59)
    at org.scalajs.jsenv.ExternalJSEnv$AsyncExtRunner.await(ExternalJSEnv.scala:172)
    at org.scalajs.jsenv.AsyncJSRunner$class.awaitOrStop(AsyncJSRunner.scala:76)
    at org.scalajs.jsenv.ExternalJSEnv$AsyncExtRunner.awaitOrStop(ExternalJSEnv.scala:172)
    at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6$$anonfun$apply$1.apply$mcV$sp(ScalaJSRunner.scala:96)
    at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6$$anonfun$apply$1.apply(ScalaJSRunner.scala:96)
    at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6$$anonfun$apply$1.apply(ScalaJSRunner.scala:96)
    at scala.util.Try$.apply(Try.scala:161)
    at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6.apply(ScalaJSRunner.scala:96)
    at org.scalajs.testadapter.ScalaJSRunner$$anonfun$6.apply(ScalaJSRunner.scala:96)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at org.scalajs.testadapter.ScalaJSRunner.done(ScalaJSRunner.scala:96)
    at sbt.Defaults$$anonfun$allTestGroupsTask$1$$anonfun$16.apply(Defaults.scala:590)
    at sbt.Defaults$$anonfun$allTestGroupsTask$1$$anonfun$16.apply(Defaults.scala:588)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at sbt.Defaults$$anonfun$allTestGroupsTask$1.apply(Defaults.scala:588)
    at sbt.Defaults$$anonfun$allTestGroupsTask$1.apply(Defaults.scala:586)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I thought it was due to Undertow hanging, but I have the same exact result with using Node. So it doesn't seem to be related to that. I then thought it could be due to using RösHTTP, which also depends on Monix, but removing that and using plain JSDOM to fetch pages still hangs in the end.

[ebruchez]

The timeout was due to not calling JSDOM window.close().

[ebruchez]

For reference, the integration tests we have now are on ignore() as they don't run with TravisCI. Next steps:

• [x] figure out why they don't work on TravisCI (use Travis debug mode?)

[ebruchez]

Consider:

• [ ] review Puppeteer
• [ ] review Headless Chromium
• [ ] review Travis GUI and Headless Browser Testing

[ebruchez]

• Puppeteer
  • says "Speed: Puppeteer has almost zero performance overhead over an automated page."
  • You install it via npm and we should be able to run it from a Scala.js ScalaTest test.
• Headless Chrome is supposed to work on Travis.

So the missing part is starting a server. We do have a couple of options:

• Docker
• Tomcat/Undertow/other servlet container

[ebruchez]

Puppeteer 2 is available

[avernet]

Hopefully this has changed in the last 2 years.

[ebruchez]

We probably would need to upgrade our Travis setup from the old Trusty to Xenial or Bionic.

[ebruchez]

Hopefully this has changed in the last 2 years.

Doesn't seem like it: you must either pass a JavaScript function or a string containing some JavaScript expression (or function?). Not sure how much this is needed but hopefully we can work around this.

[ebruchez]

What are the next steps for this? For the replication tests, we already have code which:

• starts and stop Docker containers running Orbeon Forms
• connects to them via JSDOM
• run code within JSDOM windows

Thoughts:

• For simple tests for which JSDOM is enough, we could set up a very similar infrastructure.
• For more complex tests, using Puppeteer is probably the way to go, but that's a little more investigation.

Steps:

• [x] decide on first simple test(s)
• [ ] implement JSDOM infrastructure
• [x] write first test
• [ ] decide on more complex test benefitting from Puppeteer
• [ ] implement Puppeteer infrastructure
• [ ] write first test

[ebruchez]

With #5973, we now have now some tests that:

• run Form Runner in Docker
• runs browser tests using JSDOM connecting to that Form Runner instance

For more tests, it would be good to not have to restart the Docker container with Tomcat, as that's very costly.