Using jBrowserDriver under Scala in sbt test

[MaggieLeber]

Anybody got a clue on this one? sbt does create a rather unique classpath/classloader environment, but I was hoping to simply swap out our use of HtmlUnit with jBrowserDriver

I think this is the relevant stackdump segment...

Caused by: java.lang.NullPointerException at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:354) at java.util.ServiceLoader$LazyIterator.access$600(ServiceLoader.java:323) at java.util.ServiceLoader$LazyIterator$1.run(ServiceLoader.java:396) at java.util.ServiceLoader$LazyIterator$1.run(ServiceLoader.java:395) at java.security.AccessController.doPrivileged(Native Method) at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:398) at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474) at io.github.lukehutch.fastclasspathscanner.classpath.ClasspathFinder.parseSystemClasspath(ClasspathFinder.java:319) at io.github.lukehutch.fastclasspathscanner.classpath.ClasspathFinder.getUniqueClasspathElements(ClasspathFinder.java:375) at com.machinepublishers.jbrowserdriver.JBrowserDriver.<clinit>(JBrowserDriver.java:128) at common.Common$class.$init$(Common.scala:54)...

[hollingsworthd]

We're using what's now an older version of fast-classpath-scanner. If you could try building with the latest that might help diagnose. I don't know how sbt resolves dependencies but try just adding:

libraryDependencies += "io.github.lukehutch" % "fast-classpath-scanner" % "1.93.0"

[MaggieLeber]

Thanks, I'll give it a try.

[MaggieLeber]

With Luke's 1.93.0 I'm getting

Caused by: java.lang.ClassNotFoundException: io.github.lukehutch.fastclasspathscanner.classpath.ClasspathFinder
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.machinepublishers.jbrowserdriver.JBrowserDriver.<clinit>(JBrowserDriver.java:128)

[hollingsworthd]

Looks like the API changed. This project will need to be updated.

[lukehutch]

Hi guys, yeah, sorry, the 1.9x.y versions are in flux (they're basically the beta series for 2.0), but things are fast approaching stability.

I'm curious though, why are you using introspection to call internal classes within FastClasspathScanner, rather than using the public API? Or is something else going on?

[hollingsworthd]

I think that's just sbt making the stack trace confusing. We call new ClasspathFinder().getUniqueClasspathElements()

[lukehutch]

Ah, I see. That's not the public API for this. You should be calling FastClasspathScanner#getUniqueClasspathElements(). ClasspathFinder() is an internal class. In fact, as of a few minor releases ago, the class is marked as package-private. The mechanism for finding unique classpath elements has also changed quite a lot, because it is now parallelized like the actual scanning process. The FastClasspathScanner class sends ClasspathFinder some required parameters, like an ExecutorService.

See:

https://github.com/lukehutch/fast-classpath-scanner/wiki/1.-Usage#listing-classpath-elements

[hollingsworthd]

Thanks, Luke. Yes we were on v1.9.19 when that worked. It's updated now to the latest.

@MaggieLeber Can you retry this with version 0.16.2 of this project?

[lukehutch]

Thanks. FYI, I'm probably also going to add ScanResult#getUniqueClasspathElements(), so that you can read the classpath elements from the scan result after a scan. This may eliminate duplication of work, if you need to both get the unique classpath elements and perform a scan, since a scan needs to get the unique classpath elements too.

[MaggieLeber]

@hollingsworthd sure... should get to it sometime this weekend.

[MaggieLeber]

Well, can't say I've ever seen this error before :-) (this is with "com.machinepublishers" % "jbrowserdriver" % "0.16.2" % "test" in libraryDependencies and letting it pick up whatever sub-dependencies Ivy/Maven thinks it needs.)

 java.lang.ExceptionInInitializerError
       at com.machinepublishers.jbrowserdriver.JBrowserDriverServer.main(JBrowserDriverServer.java:74)
 Caused by: java.lang.SecurityException: class "org.apache.http.cookie.CookieSpecProvider"'s signer information does not match signer information of other classes in the same package
       at java.lang.ClassLoader.checkCerts(ClassLoader.java:895)
       at java.lang.ClassLoader.preDefineClass(ClassLoader.java:665)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:758)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
       at com.machinepublishers.jbrowserdriver.CookieStore.<clinit>(CookieStore.java:42)

I'm going to poke around the dependencies tree to see why I've got this situation:

[hollingsworthd]

Might have been that this project had a custom class with an Apache package name. Version 0.16.3 of this resolves that.

[MaggieLeber]

This still seems to be failing the same way (the SecurityException) with jBrowserDriver 0.16.3. I seem to have an embarrassing number of httpclient versions floating around in this project, which I'm still trying to suss out.

[MaggieLeber]

Very odd indeed. I still crash trying to 'new' up a jBrowserDriver, with:

[2016-08-01T17:34:07.597] java.lang.ExceptionInInitializerError
[2016-08-01T17:34:07.598]       at com.machinepublishers.jbrowserdriver.JBrowserDriverServer.main(JBrowserDriverServer.java:74)
[2016-08-01T17:34:07.598] Caused by: java.lang.SecurityException: class "org.apache.http.cookie.CookieSpecProvider"'s signer information does not match signer information of other classes in the same package

but when I run in debug with a trap set for SecurityException it seems to be a different issue altogether early on:

yet if I turn it loose to run after the trap, I still get the "signer information doesn't match" stackdump...

[MaggieLeber]

Going bonkers trying to track down failure...just realized the SecurityException I'm trapping has nothing to do with the CookieSpecProvider problem because that's happening not only in a completely different thread, it's in a separate JVM altogether that my debugger isn't talking to D'oh!.

[MaggieLeber]

I just wanna say that these obfuscated jar names in jbdclasspath are simply a pure ^%&^%)(_$ joy to work with... :-)

[hollingsworthd]

They're not intentionally obfuscated. It unpacks jars within jars (which many apps have but isn't supported by default class loaders) and gives them a random name. It was a little easier to write the code this way to not have to worry about naming conflicts or filename restrictions cross-platform. JBrowserDriver.java around line 132 is where this happens.

[hollingsworthd]

For anything that's not a jar within a jar, it just references where the jar already is on the filesystem in the manifest file of the generated jar. Overall the goal is just that the child process gets the same classpath as the parent process.

Probably in this reshuffling of the classpath some signature file for a signed jar gets in the wrong place. It's probably a bug in this project and not necessarily something wrong with how sbt build is configured. Signature files are usually in META-INF/.SF, META-INF/.DSA, META-INF/*.RSA

[MaggieLeber]

Yeah, my debugger and I were there (line 132) today :-) Along with the two jars in my codebase bearing signatures from Amazon that expired in 2012... :-) I ended up needing to unjar them to find out what they used to be. :-) When I remove the signatures (with a cute little gadget from Servoy Stuff called CodeSigner) while holding a breakpoint before the process launches, I get an error that looks a lot like the symptom of #171

So me and the sbt whatDependsOn and dependencyGraphMl are getting to be good friends, along with the 394 nodes (which include no fewer than six different versions of HttpClient) and 969 edges in my dependency graph. :-)

I'll get there. It's just going to take time.

[MaggieLeber]

Thinking there's gotta be a better way to satisfy jBrowserDriver's dependencies than cloning the classpath of the host system. If only for performance reasons...

[hollingsworthd]

I'm open to suggestions (and pull requests). The problem is there are any number of ways to package this project and its dependencies. The only way I know of to satisfy the classpath aside from how it's currently done is to package all the dependencies into the jar of this project. That sort of flies in the face of how the whole Java/Maven ecosystem works.

Performance-wise it's not really an issue. The cost is loading classes and that's only done on demand. I could be wrong but I don't think extraneous items on the classpath add much overhead--classes that aren't used aren't loaded. And the classpath jar is only generated once.

There is one alternative which would basically be to let people specify their own classpath and dependencies. And at that point, really the entirety of the logic to load the child process should just be made pluggable. But that's a large change; it's probably just easier to get whatever rough spots are in the classpath code worked out.

[MaggieLeber]

I was thinking about the cost of classpath searches...maybe that's not a big deal. My use case is running a test suite that has all the classes necessary to run a server on his classpath, and the current scheme makes a facsimile of that classpath for a separate process who I would think is never going to need the serverside classes. Most of our stuff doesn't compile with javac or build with Maven, so I guess we're an edge case from that point of view. Let me think on possibilites; I may eventually have a PR for you if I'm struck with something brilliant.

[MaggieLeber]

I'd been laboring under the impression that JBD was copying my entire classpath for the process to use. But looking at it closely, what I'm actually getting is an empty classpath.jar and 16 sibling random-named jars.

Interestingly, when I set up to build JBD from source, I note that the nine declared dependencies

seem to expand into 17 total jars

C:\Users\maggie\.m2\repository\com\sun\jna\jna\3.0.9\jna-3.0.9.jar
C:\Users\maggie\.m2\repository\commons-codec\commons-codec\1.10\commons-codec-1.10.jar
C:\Users\maggie\.m2\repository\commons-io\commons-io\2.2\commons-io-2.2.jar
C:\Users\maggie\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar
C:\Users\maggie\.m2\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar
C:\Users\maggie\.m2\repository\io\github\lukehutch\fast-classpath-scanner\1.93.0\fast-classpath-scanner-1.93.0.jar
C:\Users\maggie\.m2\repository\org\apache\httpcomponents\httpclient\4.5.2\httpclient-4.5.2.jar
C:\Users\maggie\.m2\repository\org\apache\httpcomponents\httpclient-cache\4.5.2\httpclient-cache-4.5.2.jar
C:\Users\maggie\.m2\repository\org\apache\httpcomponents\httpcore\4.4.4\httpcore-4.4.4.jar
C:\Users\maggie\.m2\repository\org\seleniumhq\selenium\selenium-api\2.53.0\selenium-api-2.53.0.jar
C:\Users\maggie\.m2\repository\org\seleniumhq\selenium\selenium-remote-driver\2.53.0\selenium-remote-driver-2.53.0.jar
C:\Users\maggie\.m2\repository\org\seleniumhq\selenium\selenium-server\2.53.0\selenium-server-2.53.0.jar
C:\Users\maggie\.m2\repository\org\slf4j\slf4j-api\1.7.2\slf4j-api-1.7.2.jar
C:\Users\maggie\.m2\repository\org\slf4j\slf4j-simple\1.7.2\slf4j-simple-1.7.2.jar
C:\Users\maggie\.m2\repository\org\zeroturnaround\zt-exec\1.7\zt-exec-1.7.jar
C:\Users\maggie\.m2\repository\org\zeroturnaround\zt-process\1.3\zt-process-1.3.jar

I can't imagine that's even close to what's on my classpath during sbt testOnly

Next I may put a patch on JBD to use the actual jar names vice the random names so I can get a clearer picture of what's going on here...

[hollingsworthd]

Only jars within jars are copied to the tmp dir. The standard classloader can't actually load jars within jars so they must be extracted. The rest of the classpath is merely referenced in the manifest.mf file inside classpath.jar.

[MaggieLeber]

Ah...I knew it must be good for something.

I just realized I can pass JVM settings to the launched process, so I can set a separate debugger port and see what's actually going on.

Duh.

[MaggieLeber]

Finally broke into the ClassLoader that's complaining.

Judging by the manifests somehow there's an httpcore library at the 4.1 level in here. Nothing in my sbt project will fess up to it, and 4.1 is antique.

[pointflow] $ whatDependsOn org.apache.httpcomponents httpcore 4.1
[error] Expected '4.4-beta1'
[error] Expected '4.4.4'
[error] Expected '4.2.4'
[error] Expected '4.0'
[error] Expected '4.0.1'

[hollingsworthd]

If it's excluded I think it will work. There could be some enhancements to make debugging this classpath issues easier. See #181 for future status on that. I think this issue can be closed given that FastClasspathScanner is now updated. Thanks for raising this issue!

[MaggieLeber]

It is excluded, from Sbt's point of view. No idea where it's coming from, it's not in my Ivy cache.

[hollingsworthd]

If you comment out lines 124-140 in JBrowserDriver.java which are responsible for extracting jars within a jar and it works, then one of the dependencies has included httpcore within itself and isn't governed by the package management system. That might be a bug with that respective project. It could also be an enhancement to disable this jar-in-jar support with this project. But if commenting out those lines doesn't work, then it's coming from some dependency in the dependency management system or your build scripts.

[hollingsworthd]

v0.16.4 is released. If the process fails to launch the first try, it will retry automatically with jar-in-jar support disabled.

[MaggieLeber]

I like 0.16.4 ...I did have to warn testers that it still stackdumps the signing error.

However, when deployed to our codebase on Tuesday,, everybody who runs on Macs ran out of file handles when they tried to run sbt builds.

[2016-08-09T19:13:53.531] Caused by: java.io.FileNotFoundException: /Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home/jre/lib/ext/localedata.jar (Too many open files in system)
[2016-08-09T19:13:53.531]   at java.util.zip.ZipFile.open(Native Method)
[2016-08-09T19:13:53.531]   at java.util.zip.ZipFile.<init>(ZipFile.java:220)
[2016-08-09T19:13:53.532]   at java.util.zip.ZipFile.<init>(ZipFile.java:150)
[2016-08-09T19:13:53.532]   at java.util.jar.JarFile.<init>(JarFile.java:166)
[2016-08-09T19:13:53.532]   at java.util.jar.JarFile.<init>(JarFile.java:103)
[2016-08-09T19:13:53.532]   at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:893)
[2016-08-09T19:13:53.532]   at sun.misc.URLClassPath$JarLoader.access$700(URLClassPath.java:756)
[2016-08-09T19:13:53.533]   at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:838)
[2016-08-09T19:13:53.533]   at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:831)
[2016-08-09T19:13:53.533]   at java.security.AccessController.doPrivileged(Native Method)
[2016-08-09T19:13:53.533]   at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:830)
[2016-08-09T19:13:53.533]   at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1001)
[2016-08-09T19:13:53.534]   ... 19 more
[trace] Stack trace suppressed: run last workflow/test:testOnly for the full output.
[error] Could not run test MasterSuite: org.openqa.selenium.WebDriverException: Could not launch browser.

Clearly we need a better way to build a classpath containing only the jars that JBD actually needs. The JBD process is a client and I don't see the use case for him having all the server jars. It's mostly harmless, except when it causes havok. ;-)

I'll think about this in my copious spare time. :-)

[hollingsworthd]

I think the way to do it would be a PR to FastClasspathScanner. FastClasspathScanner takes a whitelist when it's constructed (or a blacklist) but there's no way restrict it to only the whitelist (or to blacklist everything). If it could restrict to only whitelist items then we could pass to its constructor:

      "com.google.common",
      "com.sun.jna",
      "com.thoughtworks.selenium",
      "org.openqa.grid",
      "org.openqa.selenium",
      "org.apache.commons.codec",
      "org.aapche.commons.io",
      "org.apache.commons.lang",
      "org.apache.commons.logging",
      "org.apache.http",
      "org.slf4j",
      "org.zeroturnaround.process",
      "org.zeroturnaround.exec"

[MaggieLeber]

Interesting idea. Adding it to copious free-time thoughts.

[hollingsworthd]

Ditto