search

sscarduzio / elasticsearch-readonlyrest-plugin

Free Elasticsearch security plugin and Kibana security plugin: super-easy Kibana multi-tenancy, Encryption, Authentication, Authorization, Auditing
https://readonlyrest.com
GNU General Public License v3.0
953 stars 166 forks source link

JarHell on integration tests (testcontainers) #184

Closed sscarduzio closed 7 years ago

sscarduzio commented 7 years ago

The below error happens when calling ./gradlew check.

checking for jar hell...
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: com.google.thirdparty.publicsuffix.PublicSuffixPatterns
jar1: /Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/21.0/3a3d111be1be1b745edfa7d91678a12d7ed38709/guava-21.0.jar
jar2: /Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.testcontainers/testcontainers/1.1.9/393068ce7bcbf8b8d8a0b076aff538baf952258b/testcontainers-1.1.9.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:187)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:88)
    at org.elasticsearch.bootstrap.JarHell.main(JarHell.java:70)
sscarduzio commented 7 years ago

Managed to shadow the whole guava, but I have another collision on junixsocket. Probably testcontainers can shadow that too?

@coutoPL do you think this would go anywhere? There would be obvious benefits to keep testcontainers in the main repo, i.e. your tests actually depend on classes of the internal API of ReadonlyREST. If we were to create a new sbt project we'd have the same issue, wouldnt we?

Not sure if we would, this JarHell detection is something in ES build tool. Right?

checking for jar hell...
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: org.newsclub.net.unix.AFUNIXSocketImpl$1
jar1: /Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.testcontainers/testcontainers/1.1.9/393068ce7bcbf8b8d8a0b076aff538baf952258b/testcontainers-1.1.9.jar
jar2: /Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.kohlschutter.junixsocket/junixsocket-common/2.0.4/b4d1870bf903412533e0b79c6fcd402defcfc05b/junixsocket-common-2.0.4.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:187)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:88)
    at org.elasticsearch.bootstrap.JarHell.main(JarHell.java:70)
:jarHell FAILED
coutoPL commented 7 years ago

jarhell is fired in: esbootstrap class, esbootstrap test class and as one of subtask in build task. Yesterday I found a solution how to disable jarhell subtask. This is enough for us.

sscarduzio commented 7 years ago

do you think we'll get issues for disabling jarhell check? I mean it's firing because collisions are found.. What do you think?

coutoPL commented 7 years ago

it would be perfect if we could disable jarhell check only for tests (with assume, that we are not going to use ES integration test class). Sadly, I don't know how to do it. But I think the check will be performed at ES startup (see: https://github.com/elastic/elasticsearch/blob/aad51d44ab7363e53da773afa4707e0165eebda0/core/src/main/java/org/elasticsearch/bootstrap/Bootstrap.java#L206).

sscarduzio commented 7 years ago

makes sense, how did you manage to disable it globally?

coutoPL commented 7 years ago

but one thing is strange ... the failing check is between guava and testcontainers. The second one should not be a part of output jar. So, this jar hell has no chance to cause collision at plugin runtime. So, why tests dependencies are checked with main dependencies?

coutoPL commented 7 years ago

jarHell.enabled=false in build.gradle

sscarduzio commented 7 years ago

how about we just disable this thing in the build system, as it's checked runtime whenever you run ES? It sounds bad as I said it, but in fact integration tests will catch jar hells for us from within docker when they run a whole ES in the container.

coutoPL commented 7 years ago

yes, this is some kind of solution. I've started to write base integration test which will create and run container with ES (and readonlyrest plugin) and other containers with external dependencies (eg. LDAP), and allow to interact with ES using HTTP. As you said, if somehow jarhell occurs at runtime we're able to know it thanks to integration tests.

sscarduzio commented 7 years ago

Disabled Jar hell check, getting Jar Hell related shitstorm and tests crash:

  2> Exception in thread "Thread-2" java.lang.ExceptionInInitializerError
  2>    at org.newsclub.net.unix.AFUNIXSocket.<init>(AFUNIXSocket.java:36)
  2>    at org.newsclub.net.unix.AFUNIXSocket.newInstance(AFUNIXSocket.java:54)
  2>    at org.rnorth.tcpunixsocketproxy.TcpToUnixSocketProxy.lambda$start$0(TcpToUnixSocketProxy.java:86)
  2>    at java.lang.Thread.run(Thread.java:745)
  2> Caused by: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException
  2>    at org.newsclub.net.unix.NativeUnixSocket.<clinit>(NativeUnixSocket.java:42)
  2>    ... 4 more

This is the unix sockets lib that caused the jar hell in the first place (once you shadowed guava). Do you think we could shadow this dependency too as a pull request to testcontainers?

sscarduzio commented 7 years ago

I shadowed that shit too, but we have other collision between testcontainers' dependencies' dependencies which means you can't shadow them from testcontainers' pom.

Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: org.apache.commons.logging.impl.NoOpLog
jar1: /Users/sscarduzio/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar
jar2: /Users/sscarduzio/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.21/jcl-over-slf4j-1.7.21.jar
    at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:187)
    at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:88)
    at org.elasticsearch.bootstrap.JarHell.main(JarHell.java:70)

Ideally we should shadow stuff from our gradle project like testcontainers does in Maven. Looks simple, but I couldn't manage. I have no idea why it's so difficult for me to convince Gradle to do anything. :rage4: :rage4: 😫

coutoPL commented 7 years ago

I've found a workaround to invoke JarHell check and have tests with testcontainers. I'll finish base integration test and create PR. Stay tuned :)

sscarduzio commented 7 years ago

Awesome, can't wait!

sscarduzio commented 7 years ago

Hey @coutoPL is there anything I can help with this?

coutoPL commented 7 years ago

I didn't have much time this week, but today evening I'm going to back to the issue.

coutoPL commented 7 years ago

Ok, done. Gradle sucks! https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/pull/190

sscarduzio commented 7 years ago

Do you use a Linux in your machine? I'm having crazy issues with unix sockets https://github.com/docker-java/docker-java/issues/537

coutoPL commented 7 years ago

Yes, I've noticed that Docker for Mac has problem with this one test. But only when I was running tests from console - when I had run it in intellij, it worked. And then have checked at machine with Linux and everything seemed to be ok. Travis said it is ok too, so I gave up.

sscarduzio commented 7 years ago

You're right. It works in IntelliJ. What the hell, I have no idea why. Fair enough for now.

sscarduzio commented 7 years ago

@coutoPL I'm porting the latest changes to 5.1.x any idea about why LDAP integration test is failing/timeout-ing? https://travis-ci.org/sscarduzio/elasticsearch-readonlyrest-plugin/jobs/208190700

coutoPL commented 7 years ago

Is this deterministic behaviour? I know this is stupid, but maybe 1 minute timeout to start container is not enough.

sscarduzio commented 7 years ago

I tried 3 times + multiple times in my laptop. Now I raised the timeout to 180s and get another error

/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java -ea -Didea.launcher.port=7532 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/tools.jar:/me/elasticsearch-readonlyrest-plugin-5.x/build/classes/test:/me/elasticsearch-readonlyrest-plugin-5.x/build/resources/test:/me/elasticsearch-readonlyrest-plugin-5.x/build/classes/main:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/21.0/3a3d111be1be1b745edfa7d91678a12d7ed38709/guava-21.0.jar:/me/elasticsearch-readonlyrest-plugin-5.x/build/generated-resources:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.elasticsearch.plugin/transport-netty4-client/5.1.2/7e5c397f6da0ffde15e05e4b3ead61d96176d9a1/transport-netty4-client-5.1.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.unboundid/unboundid-ldapsdk/3.2.0/f76725e5a215ea468ecda06a8d66a809281e685f/unboundid-ldapsdk-3.2.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.elasticsearch/elasticsearch/5.1.2/d949afc1ae55b42bf94bffff6964727cf173340e/elasticsearch-5.1.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.locationtech.spatial4j/spatial4j/0.6/21b15310bddcfd8c72611c180f20cf23279809a3/spatial4j-0.6.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.vividsolutions/jts/1.13/3ccfb9b60f04d71add996a666ceb8902904fd805/jts-1.13.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.7/8de00e382a817981b737be84cb8def687d392963/log4j-api-2.7.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.7/a3f2b4e64c61a7fc1ed8f1e5ba371933404ed98a/log4j-core-2.7.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna/4.2.2/5012450aee579c3118ff09461d5ce210e0cdc2a9/jna-4.2.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.elasticsearch.test/framework/5.1.2/62dd849709b087d490492bd67da4e624864abf10/framework-5.1.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-all/1.10.19/539df70269cc254a58cccc5d8e43286b4a73bf30/mockito-all-1.10.19.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/junit/junit/4.12/2973d150c0dc1fefe998f834810d68f278ea58ec/junit-4.12.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.6.Final/dc1a304ee7c448bfa241f60c41948db6f12b653b/netty-buffer-4.1.6.Final.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.gradle/gradle-tooling-api/2.13/4a49ebf6bcf841d549e1d3b6c0cca8f422606f48/gradle-tooling-api-2.13.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.6.Final/86ce3c559af6759a0b991b9ad2faec20981e18e/netty-codec-4.1.6.Final.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.6.Final/ee6c65cb3f39f212241e9131dc05a837e776fd70/netty-codec-http-4.1.6.Final.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.testcontainers/testcontainers/1.1.9/393068ce7bcbf8b8d8a0b076aff538baf952258b/testcontainers-1.1.9.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.6.Final/654e4d8e87bc442be5c1d82d0369a3885f7168ea/netty-common-4.1.6.Final.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.6.Final/a24e92d67e467a3b97ff63642ab56637990685fd/netty-handler-4.1.6.Final.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.6.Final/2a1c97445b6aa544cd029faa72abfb3f85e9957e/netty-resolver-4.1.6.Final.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.6.Final/4cba91ad2711c13a063d480f01edf70aa3f9387d/netty-transport-4.1.6.Final.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-core/6.3.0/d3c87ea89e2f83e401f9cc7f14e4c43945f7f1e1/lucene-core-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-analyzers-common/6.3.0/494aed699af238c3872a6b65e17939e9cb7ddbe0/lucene-analyzers-common-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-backward-codecs/6.3.0/77dede7dff1b833ca2e92d8ab137edb209354d9b/lucene-backward-codecs-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-grouping/6.3.0/2c96d59e318ea66838aeb9c5cfb8b4d27b40953c/lucene-grouping-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-highlighter/6.3.0/4f154d8badfe47fe45503c18fb30f2177f758794/lucene-highlighter-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-join/6.3.0/79b898117dcfde2981ec6806e420ff218842eca8/lucene-join-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-memory/6.3.0/89edeb404e507d640cb13903acff6953199704a2/lucene-memory-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-misc/6.3.0/2d0e1f5a9df15ac911ad495bad5ea253ab50a9f/lucene-misc-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queries/6.3.0/eb7938233c8103223069c7b5b5f785b4d20ddafa/lucene-queries-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-queryparser/6.3.0/e979fb02155cbe81a8d335d6dc41d2ef06be68b6/lucene-queryparser-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-sandbox/6.3.0/257387c45c6fa2b77fd6931751f93fdcd798ced4/lucene-sandbox-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial/6.3.0/3cf5fe5402b5e34b240b73501c9e97a82428259e/lucene-spatial-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial-extras/6.3.0/1b77ef3740dc885c62d5966fbe9aea1199d344fb/lucene-spatial-extras-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-spatial3d/6.3.0/aa94b4a8636b3633008640cc5155ad354aebcea5/lucene-spatial3d-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-suggest/6.3.0/ed5d8ee5cd7edcad5d4ffca2b4540ccc844e9bb0/lucene-suggest-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securesm/1.1/1e423447d020041534be94c0f31a49fbdc1f2950/securesm-1.1.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.2/98cafc6081d5632b61be2c9e60650b64ddbc637c/jopt-simple-5.0.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.carrotsearch/hppc/0.7.1/8b5057f74ea378c0150a1860874a3ebdcb713767/hppc-0.7.1.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/joda-time/joda-time/2.9.5/5f01da7306363fad2028b916f3eab926262de928/joda-time-2.9.5.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.15/3b132bea69e8ee099f416044970997bde80f4ea6/snakeyaml-1.15.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.8.1/fd13b1c033741d48291315c6370f7d475a42dccf/jackson-core-2.8.1.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-smile/2.8.1/5b73867bc12224946fc67fc8d49d9f5e698d7f/jackson-dataformat-smile-2.8.1.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.8.1/eb63166c723b0b4b9fb5298fca232a2f6612ec34/jackson-dataformat-yaml-2.8.1.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.8.1/3a6fb7e75c9972559a78cf5cfc5a48a41a13ea40/jackson-dataformat-cbor-2.8.1.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.tdunning/t-digest/3.0/84ccf145ac2215e6bfa63baa3101c0af41017cfc/t-digest-3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.6/7495feb7f71ee124bd2a7e7d83590e296d71d80e/HdrHistogram-2.1.6.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.carrotsearch.randomizedtesting/randomizedtesting-runner/2.4.0/222eb23dd6f45541acf6a5ac69cd9e9bdce25d2/randomizedtesting-runner-2.4.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-all/1.3/63a21ebc981131004ad02e0434e799fd7f3a8d5a/hamcrest-all-1.3.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-test-framework/6.3.0/a6ad70bafbabbc82830f7e0b1d6ac1f4d74831d7/lucene-test-framework-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.lucene/lucene-codecs/6.3.0/8e58e160a4751200987e60a365f4370d88fd9942/lucene-codecs-6.3.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.elasticsearch.client/rest/5.1.2/8251790487394792e3e13e20225e36edf15cb474/rest-5.1.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.2/733db77aa8d9b2d68015189df76ab06304406e50/httpclient-4.5.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.5/e7501a1b34325abb00d17dde96150604a0658b54/httpcore-4.4.5.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.1.3/f6f66e966c70a83ffbdb6f17a0919eaf7c8aca7f/commons-logging-1.1.3.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.10/4b95f4897fa13f2cd904aee711aeafc0c5295cd8/commons-codec-1.10.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.elasticsearch/securemock/1.2/98201d4ad5ac93f6b415ae9172d52b5e7cda490e/securemock-1.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpasyncclient/4.1.2/95aa3e6fb520191a0970a73cf09f62948ee614be/httpasyncclient-4.1.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore-nio/4.4.5/f4be009e7505f6ceddf21e7960c759f413f15056/httpcore-nio-4.4.5.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.21/139535a69a4239db087de9bab0bee568bf8e0b70/slf4j-api-1.7.21.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.javassist/javassist/3.18.1-GA/d9a09f7732226af26bf99f19e2cffe0ae219db5b/javassist-3.18.1-GA.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.kohlschutter.junixsocket/junixsocket-common/2.0.4/b4d1870bf903412533e0b79c6fcd402defcfc05b/junixsocket-common-2.0.4.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/log4j/log4j/1.2.17/5af35056b4d257e4b64b9e8069c0746e8b08629f/log4j-1.2.17.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/com.kohlschutter.junixsocket/junixsocket-native-common/2.0.4/726bd66a934dea39c817382986496fa4eda96411/junixsocket-native-common-2.0.4.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.scijava/native-lib-loader/2.0.2/1451fa03954c5e31a358b411147de472b4dab92c/native-lib-loader-2.0.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-compress/1.12/84caa68576e345eb5e7ae61a0e5a9229eb100d7b/commons-compress-1.12.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/commons-lang/commons-lang/2.6/ce1edb914c94ebc388f086c6827e8bdeec71ac2/commons-lang-2.6.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.5/2852e6e05fbb95076fc091f6d1780f1f8fe35e0f/commons-io-2.5.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.slf4j/jcl-over-slf4j/1.7.21/331b564a3a42f002a0004b039c1c430da89062cd/jcl-over-slf4j-1.7.21.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.rnorth/tcp-unix-socket-proxy/1.0.1/72ff1904b2db1cdfc2e9ef0a61be0ee6f9e4736c/tcp-unix-socket-proxy-1.0.1.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.zeroturnaround/zt-exec/1.8/7d6c8d9f6067870cf6822741ad2e8a8fea37bbc1/zt-exec-1.8.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.rnorth.duct-tape/duct-tape/1.0.6/849cdd677ccd969f7a8810294dfe4c7c6dbef308/duct-tape-1.0.6.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-ext/1.7.2/676a2fc95740d6d070e2c7c54920e83e6c6969ec/slf4j-ext-1.7.2.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/ch.qos.cal10n/cal10n-api/0.7.4/132b7e1702af0335cf4259d30aaf959264db688f/cal10n-api-0.7.4.jar:/Users/sscarduzio/.gradle/caches/modules-2/files-2.1/org.hamcrest/hamcrest-core/1.3/42a25dc3219429f0e5d060061f71acb49bf010a0/hamcrest-core-1.3.jar" com.intellij.rt.execution.application.AppMain com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 org.elasticsearch.plugin.readonlyrest.integration.LdapIntegrationTests
Mar 06, 2017 2:30:57 PM org.elasticsearch.plugin.readonlyrest.utils.containers.LdapContainer create
INFO: Creating LDAP container ...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
log4j:WARN No appenders could be found for logger (org.testcontainers.shaded.io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Mar 06, 2017 2:31:19 PM org.elasticsearch.plugin.readonlyrest.utils.containers.LdapContainer create
INFO: Creating LDAP container ...
Mar 06, 2017 2:31:20 PM org.elasticsearch.plugin.readonlyrest.utils.containers.LdapContainer$1 waitUntilReady
INFO: Waiting for LDAP container ...

java.lang.ExceptionInInitializerError
    at sun.misc.Unsafe.ensureClassInitialized(Native Method)
    at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
    at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:142)
    at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
    at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
    at java.lang.reflect.Field.get(Field.java:393)
    at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
    at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
    at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
    at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
    at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:179)
    at org.elasticsearch.plugin.readonlyrest.utils.containers.MultiContainer.lambda$new$1(MultiContainer.java:41)
    at java.util.ArrayList.forEach(ArrayList.java:1249)
    at org.elasticsearch.plugin.readonlyrest.utils.containers.MultiContainer.<init>(MultiContainer.java:41)
    at org.elasticsearch.plugin.readonlyrest.utils.containers.MultiContainer$Builder.build(MultiContainer.java:53)
    at org.elasticsearch.plugin.readonlyrest.integration.LdapIntegrationTests.<clinit>(LdapIntegrationTests.java:44)
    ... 22 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:83)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:172)
    ... 27 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:245)
    at org.testcontainers.containers.GenericContainer.lambda$start$0(GenericContainer.java:174)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
    ... 28 more
Caused by: java.lang.IllegalStateException: LDAPException(resultCode=81 (server down), errorMessage='The connection is not established.')
    at org.elasticsearch.plugin.readonlyrest.utils.containers.LdapContainer$1.waitUntilReady(LdapContainer.java:106)
    at org.testcontainers.containers.GenericContainer$AbstractWaitStrategy.waitUntilReady(GenericContainer.java:901)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:450)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:224)
    ... 30 more
Caused by: LDAPException(resultCode=81 (server down), errorMessage='The connection is not established.')
    at com.unboundid.ldap.sdk.LDAPConnection.registerResponseAcceptor(LDAPConnection.java:4476)
    at com.unboundid.ldap.sdk.AddRequest.processAsync(AddRequest.java:1079)
    at com.unboundid.ldap.sdk.AddRequest.process(AddRequest.java:1010)
    at com.unboundid.ldap.sdk.LDAPConnection.add(LDAPConnection.java:1950)
    at org.elasticsearch.plugin.readonlyrest.utils.containers.LdapContainer$1.initLdap(LdapContainer.java:139)
    at org.elasticsearch.plugin.readonlyrest.utils.containers.LdapContainer$1.waitUntilReady(LdapContainer.java:104)
    ... 33 more

Test ignored.

Test ignored.

Test ignored.

Test ignored.

Process finished with exit code 255
sscarduzio commented 7 years ago

I had raised CONTAINER_STARTUP_TIMEOUT in LdapContainer and ESWithReadonlyRestContainer

coutoPL commented 7 years ago

this is weird ... log above shows that LDAP container did not start. On the other hand, log from travis shows that ES container startup failed due to timeout.

sscarduzio commented 7 years ago

OK there's something wrong with ES configuration, I tailed docker logs of the spawned ES container.

 MET:GET, PTH:_cluster/health, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,User-Agent,Host,Content-Length, EFF:0, HIS:[CONTAINER ADMIN->[auth_key->true]], [CONTAINER ADMIN->[]] } matched block: CONTAINER ADMIN match: true}
[2017-03-06T13:52:06,285][ERROR][o.e.x.m.c.i.IndexStatsCollector] [rY0RRfc] collector [index-stats-collector] - failed collecting data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:161) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:47) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:256) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:234) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:79) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.ingest.IngestActionFilter.apply(IngestActionFilter.java:82) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.plugin.readonlyrest.IndexLevelActionFilter.apply(IndexLevelActionFilter.java:106) ~[?:?]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:87) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:75) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:64) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.xpack.security.InternalClient.doExecute(InternalClient.java:82) ~[?:?]
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1226) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:69) ~[elasticsearch-5.1.2.jar:5.1.2]
    at org.elasticsearch.xpack.monitoring.collector.indices.IndexStatsCollector.doCollect(IndexStatsCollector.java:78) ~[x-pack-5.1.2.jar:5.1.2]
    at org.elasticsearch.xpack.monitoring.collector.AbstractCollector.collect(AbstractCollector.java:92) [x-pack-5.1.2.jar:5.1.2]
    at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.collect(AgentService.java:238) [x-pack-5.1.2.jar:5.1.2]
    at org.elasticsearch.xpack.monitoring.AgentService$ExportingWorker.run(AgentService.java:205) [x-pack-5.1.2.jar:5.1.2]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92-internal]

Went away with monitoring.enabled: false in the yml config file. Still hangs tho.

coutoPL commented 7 years ago

Right, docker logs can be helpful.

coutoPL commented 7 years ago

I missed last line in your comment. Still timeout occurs?

sscarduzio commented 7 years ago

Now the two ldap are started, but the ES container isnt discovered as running. I still see the requests going in the docker logs.. Not sure why the response doesnt get back to the Java stack.

[2017-03-06T14:26:01,273][INFO ][o.e.p.r.a.ACL            ] request: { ID:ae60aa92ed4e4012bc522f14bfe7f710, TYP:ClusterHealthRequest, USR:admin, BRS:true, ACT:cluster:monitor/health, OA:172.17.0.1, IDX:, MET:GET, PTH:_cluster/health, CNT:<OMITTED, LENGTH=0>, HDR:Authorization,Connection,User-Agent,Host,Content-Length, EFF:0, HIS:[CONTAINER ADMIN->[]], [CONTAINER ADMIN->[auth_key->true]] } matched block: CONTAINER ADMIN match: true}
coutoPL commented 7 years ago

Are you sure that ES 5.1.0 supports the health check used in startup strategy?

sscarduzio commented 7 years ago

interesting how I get this from the terminal while the container is up... If ES doesn't understand my request, it should return 404 or something, but not connection refused.

$ curl "http://localhost:9200/_cluster/health" curl: (7) Failed to connect to localhost port 9200: Connection refused

coutoPL commented 7 years ago

you've got connection refused because docker exposes ES 9200 on different port (this is useful if you run test in parallel). Run docker ps to find out mapped port.

sscarduzio commented 7 years ago 
1886cbd08428        testcontainers/mnbykve26kdkkwb9   "/bin/bash bin/es-..."   2 minutes ago       Up 2 minutes        0.0.0.0:32859->9200/tcp, 0.0.0.0:32858->9300/tcp   inspiring_liskov

curl "http://localhost:32859/_cluster/health"

Forbidden

sscarduzio commented 7 years ago

ah shit I forgot the auth header.

coutoPL commented 7 years ago

admin credentials are can be find here: https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/blob/master/src/test/resources/test_elasticsearch.yml#L111

sscarduzio commented 7 years ago

ok I have the info I was looking for, I printed the stack trace of the failing health checks I get a few of these:

Mar 06, 2017 3:47:27 PM org.elasticsearch.plugin.readonlyrest.utils.containers.ESWithReadonlyRestContainer$1 isReady
INFO: getting health..
org.apache.http.ConnectionClosedException: Connection closed

And then a few of these

Mar 06, 2017 3:48:02 PM org.elasticsearch.plugin.readonlyrest.utils.containers.ESWithReadonlyRestContainer$1 isReady
INFO: getting health..
java.io.IOException: listener timeout after waiting for [10000] ms

I'm afraid the health check is not hitting the right port

coutoPL commented 7 years ago

how can I reproduce it? This code is in master branch? I'll try to debug it later.

sscarduzio commented 7 years ago

it's in the 5.1 branch right now, can reproduce in travis and on my laptop (with intellij test runner)

sscarduzio commented 7 years ago

Never mind I fixed this. The test framework is solid, a piece of code was missing elsewhere in the 5.1.x wiring. Thanks a lot for the help, now I know more about the new test framework. 👍

coutoPL commented 7 years ago

Great. It means that integration tests start fulfill their role :)

sscarduzio commented 7 years ago

haha that's true :)