Can't Run v5.0.0 with Play! Scala

[piettes]

Since updating to v5.0.0, it can't get rid off warnings at compilation :

[warn] Class org.apache.lucene.queryparser.classic.QueryParser not found - continuing with a stub. [warn] Class org.apache.lucene.spatial.query.SpatialArgs not found - continuing with a stub.

And then it always crashes at runtime :

--- (Running the application, auto-reloading is enabled) ---

java.lang.NullPointerException at io.netty.channel.group.DefaultChannelGroup.add(DefaultChannelGroup.java:146) at play.core.server.NettyServer.bind(NettyServer.scala:140) at play.core.server.NettyServer.play$core$server$NettyServer$$bindChannel(NettyServer.scala:224) at play.core.server.NettyServer$$anonfun$1.apply(NettyServer.scala:216) at play.core.server.NettyServer$$anonfun$1.apply(NettyServer.scala:216) at scala.Option.map(Option.scala:146) at play.core.server.NettyServer.(NettyServer.scala:216) at play.core.server.NettyServerProvider.createServer(NettyServer.scala:279) at play.core.server.NettyServerProvider.createServer(NettyServer.scala:278) at play.core.server.DevServerStart$$anonfun$mainDev$1.apply(DevServerStart.scala:235) at play.core.server.DevServerStart$$anonfun$mainDev$1.apply(DevServerStart.scala:65) at play.utils.Threads$.withContextClassLoader(Threads.scala:21) at play.core.server.DevServerStart$.mainDev(DevServerStart.scala:64) at play.core.server.DevServerStart$.mainDevHttpMode(DevServerStart.scala:54) at play.core.server.DevServerStart.mainDevHttpMode(DevServerStart.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at play.runsupport.Reloader$.startDevMode(Reloader.scala:234) at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.devModeServer$lzycompute$1(PlayRun.scala:74) at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.play$sbt$run$PlayRun$$anonfun$$anonfun$$anonfun$$devModeServer$1(PlayRun.scala:74) at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.apply(PlayRun.scala:100) at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.apply(PlayRun.scala:53) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)

I tried to add extra dependencies in build.sbt, but I couldn't find org.apache.lucene.spatial.query.SpatialArgs

scalaVersion "2.11.8" sbt.version 0.13.12 play "2.5.9" elastic4s-core "5.0.0"

Works well with elastic4s-core 2.1.1

[sksamuel]

Does play use netty? If so is there a version conflict? Es wants 4.1.5 Are you running embedded or remote ?

[piettes]

Play uses Netty embedded. There seems to be a Version problem indeed :

[warn] There may be incompatibilities among your library dependencies. [warn] Here are some of the libraries that were evicted: [warn] * io.netty:netty-codec-http:(4.0.41.Final, 4.0.39.Final) -> 4.1.5.Final (caller: org.elasticsearch.plugin:transport-netty4-client:5.0.0, org.asynchttpclient:async-http-client:2.0.11, com.typesafe.netty:netty-reactive-streams-http:1.0.8) [warn] * io.netty:netty-handler:4.0.41.Final -> 4.1.5.Final (caller: io.netty:netty-codec-http:4.0.41.Final, org.elasticsearch.plugin:transport-netty4-client:5.0.0, com.typesafe.netty:netty-reactive-streams:1.0.8) [warn] * io.netty:netty-codec:(4.0.41.Final, 4.0.39.Final) -> 4.1.5.Final (caller: io.netty:netty-handler:4.0.41.Final, org.asynchttpclient:netty-codec-dns:2.0.11, io.netty:netty-codec-http:4.0.41.Final, org.elasticsearch.plugin:transport-netty4-client:5.0.0, io.netty:netty-handler:4.1.5.Final, io.netty:netty-codec-http:4.1.5.Final) [warn] * io.netty:netty-transport:(4.0.41.Final, 4.0.39.Final) -> 4.1.5.Final (caller: io.netty:netty-codec:4.0.41.Final, io.netty:netty-handler:4.0.41.Final, io.netty:netty-codec:4.1.5.Final, org.elasticsearch.plugin:transport-netty4-client:5.0.0, io.netty:netty-handler:4.1.5.Final, org.asynchttpclient:netty-resolver-dns:2.0.11, io.netty:netty-transport-native-epoll:4.0.41.Final) [warn] * io.netty:netty-buffer:4.0.41.Final -> 4.1.5.Final (caller: io.netty:netty-handler:4.0.41.Final, org.elasticsearch.plugin:transport-netty4-client:5.0.0, io.netty:netty-transport:4.1.5.Final, io.netty:netty-handler:4.1.5.Final, io.netty:netty-transport:4.0.41.Final, io.netty:netty-transport-native-epoll:4.0.41.Final) [warn] * io.netty:netty-common:(4.0.41.Final, 4.0.39.Final) -> 4.1.5.Final (caller: io.netty:netty-resolver:4.1.5.Final, io.netty:netty-buffer:4.0.41.Final, org.elasticsearch.plugin:transport-netty4-client:5.0.0, io.netty:netty-buffer:4.1.5.Final, org.asynchttpclient:netty-resolver:2.0.11, io.netty:netty-transport-native-epoll:4.0.41.Final) [warn] * io.netty:netty-transport-native-epoll:4.0.39.Final -> 4.0.41.Final (caller: com.typesafe.play:play-netty-server_2.11:2.5.9, org.asynchttpclient:async-http-client:2.0.11)

It needs Netty 4.0.x, ES brings 4.1.X

[piettes]

Okay I was able to make it work with some tricks :

dependencyOverrides += "io.netty" % "netty-codec-http" % "4.0.41.Final" dependencyOverrides += "io.netty" % "netty-handler" % "4.0.41.Final" dependencyOverrides += "io.netty" % "netty-codec" % "4.0.41.Final" dependencyOverrides += "io.netty" % "netty-transport" % "4.0.41.Final" dependencyOverrides += "io.netty" % "netty-buffer" % "4.0.41.Final" dependencyOverrides += "io.netty" % "netty-common" % "4.0.41.Final" dependencyOverrides += "io.netty" % "netty-transport-native-epoll" % "4.0.41.Final"

I also had to manually add lucene dependencies :

libraryDependencies += "org.apache.lucene" % "lucene-queries" % "6.2.1" libraryDependencies += "org.apache.lucene" % "lucene-queryparser" % "6.2.1" libraryDependencies += "org.apache.lucene" % "lucene-suggest" % "6.2.1" libraryDependencies += "org.apache.lucene" % "lucene-highlighter" % "6.2.1" libraryDependencies += "org.apache.lucene" % "lucene-spatial-extras" % "6.2.1" libraryDependencies += "org.apache.lucene" % "lucene-spatial3d" % "6.2.1" libraryDependencies += "org.apache.lucene" % "lucene-spatial" % "6.2.1"

[joantune]

While this is good quickfix (thanks for it) do note that If you do several queries asynchronously [or not apparently], this dependency conflict will apparently come back and bite you in the @$s. As you'll get ElasticSearch trying to access a method that does not exist in the 4.0.41.Final version of it

Exception in thread "elasticsearch[_client_][generic][T#1]" java.lang.NoSuchMethodError: io.netty.bootstrap.Bootstrap.config()Lio/netty/bootstrap/BootstrapConfig;
    at org.elasticsearch.transport.netty4.Netty4Transport.lambda$stopInternal$6(Netty4Transport.java:507)
    at org.elasticsearch.transport.netty4.Netty4Transport$$Lambda$615/1311344379.close(Unknown Source)
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
    at org.elasticsearch.common.lease.Releasables.close(Releasables.java:35)
    at org.elasticsearch.common.lease.Releasables.close(Releasables.java:45)
    at org.elasticsearch.common.lease.Releasables.close(Releasables.java:50)
    at org.elasticsearch.transport.netty4.Netty4Transport.stopInternal(Netty4Transport.java:490)
    at org.elasticsearch.transport.TcpTransport.lambda$doStop$4(TcpTransport.java:793)
    at org.elasticsearch.transport.TcpTransport$$Lambda$614/929888665.run(Unknown Source)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:444)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

[danielmorozoff]

@joantune Have you found a way to fix the method not found error on elastic?

For those interested using the akka-http could also be helpful. Here a link to a discussion https://groups.google.com/forum/#!msg/play-framework/c1yh4g-TkTE/TBhq81ZgAQAJ

[joantune]

Hi Dan! Thanks for the tip. For now i'm using Jest + es or lucene's query builders (so that I don't write them as strings), and that's it. I won't get the benefits of the transport client though. So, you're suggesting to replace Play or to simply use remote Akka actors that run with a compatible classpath?

Gee, I wish we would get Java9 already to not have these issues On Sat, Dec 10, 2016, 15:22 Dan Morozoff notifications@github.com wrote:

@joantune https://github.com/joantune Have you found a way to fix the method not found error on elastic?

For those interested using the akka-http could also be helpful. Here a link to a discussion https://groups.google.com/forum/#!msg/play-framework/c1yh4g-TkTE/TBhq81ZgAQAJ

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sksamuel/elastic4s/issues/666#issuecomment-266215548, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-DBNcx3vr9xI6p5f69NHKyLgSdf9CJks5rGsOqgaJpZM4KyS_p .

[danielmorozoff]

The link I provided leverages the experimental Akka Http provided with Play Link Basically involves some quick edits to the build.sbt file (activating akka-http and disabling netty). Do note that there are issues with this too as some overlapping dependencies do exist (they're mentioned towards the end of the discussion).

You could conceivably just use the Akka actors, but this will depend on what you're specifically looking to accomplish and how integrated you want Play to function in that.

As far as my personal attempts, I was able to get sbt to compile with the akka-http , but was having issues getting the transport layer to see ES. I'll loop back when I have more info.

Yea Java9...

[joantune]

Gotcha! Replace Netty with Akka-http on Play! Nice tip indeed, sorry for not realizing that myself by reading through the link first before asking but I'm AFK at the moment. I don't think I need to use the transport client ATM but if I do I'll try it out and let you know how it went through here. Thanks!

On Sat, Dec 10, 2016, 15:56 Dan Morozoff notifications@github.com wrote:

The link I provided leverages the experimental Akka Http provided with Play

https://www.playframework.com/documentation/2.5.x/AkkaHttpServer#Akka-HTTP-server-backend-(experimental) http://url

Basically involves some quick edits to the build.sbt file (activating akka-http and disabling netty). Do note that there are issues with this too as some overlapping dependencies do exist (they're mentioned towards the end of the discussion).

You could conceivably just use the Akka actors, but this will depend on what you're specifically looking to accomplish and how integrated you want Play to function in that.

As far as my personal attempts, I was able to get sbt to compile with the akka-http , but was having issues getting the transport layer to see ES. I'll loop back when I have more info.

Yea Java9...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sksamuel/elastic4s/issues/666#issuecomment-266217919, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-DBMfoirHHgNWpCJeqNMnXc7OcHfGEks5rGsurgaJpZM4KyS_p .

[SemanticBeeng]

Read all this careful and seem there is no production ready solution to this issue other then wrapping all ES aware code into an Akka actor and a separate module

1. the Akka-Http back end to Play! is experimental atm in Play 2.5: https://www.playframework.com/documentation/2.5.x/AkkaHttpServer

2. hacking Netty dependencies will eventually hit something like a java.lang.NoSuchMethodError

Is this a correct summary?

[joantune]

Yup AFAIK. Also, ofcourse: the Akka actor has to be run in another JVM instance

On Sun, Jan 8, 2017, 08:15 SemanticBeeng notifications@github.com wrote:

Read all this careful and seem there is no production ready solution to this issue other then wrapping all ES aware code into an Akka actor and a separate model

1.

the Akka-http back end to Play! is experimental atm in Play 2.5: https://www.playframework.com/documentation/2.5.x/AkkaHttpServer 2.

Hacking Netty dependencies will eventually hit something like a java.lang.NoSuchMethodError

Is this a correct summary?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sksamuel/elastic4s/issues/666#issuecomment-271137337, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-DBIeJVyR3zly9yGKAZeui9XFLrAEUks5rQJsygaJpZM4KyS_p .

[SemanticBeeng]

Maybe not in another JVM but another module that does not have Play! enabled

See https://github.com/adrianhurt/play-multidomain-seed/blob/master/build.sbt

If modules have separate classloaders then this should be smooth.

[sksamuel]

Elastic are eventually going to dump the transport client in favour of a REST only client. There's nothing stopping us preempting that by doing it ourselves. The elastic4s API could remain the same.

[SemanticBeeng]

Indeed that would be best. What would it take concretely?

[joantune]

I'm using the REST client as a workaround. Although a REST client is slower than transport.

ATM I'm using Jest with some of elasticsearch's libraries as per documentation so that I can build the query with programmatic query builders.

On Sun, Jan 8, 2017, 15:23 SemanticBeeng notifications@github.com wrote:

Indeed that would be best. What would it take concretely?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sksamuel/elastic4s/issues/666#issuecomment-271157447, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-DBGbqIuMg5yF7GTivGIJNbJNX-KJyks5rQP9mgaJpZM4KyS_p .

[sksamuel]

Jest is a good client. You don't get the Scala integration of course which is the main point of picking elastic4s I suppose.

[sksamuel]

@semanticbeeng would need to implement a json builder for each type of query needed to be supoorted.

[SemanticBeeng]

@sksamuel that is a serious amount of work and/but needs to be done at some point because not using REST but sockets is causing issues when IP addresses to ES are recycled...

Will try to use a different module/classloader for services using ES than the Play! module - you see any reason why that would not work?

[sksamuel]

Sounds like it should be ok using seperate classloaders. Using tcp over http is an elastic decision I just use whatever the underlying client does.

On 8 Jan 2017 4:13 p.m., "SemanticBeeng" notifications@github.com wrote:

@sksamuel https://github.com/sksamuel that is a serious amount of work and/but needs to be done at some point because not using REST but sockets is causing issues when IP addresses to ES are recycled...

Will try to use a different module/classloader for services using ES than the Play! module - you see any reason why that would not work?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sksamuel/elastic4s/issues/666#issuecomment-271160618, or mute the thread https://github.com/notifications/unsubscribe-auth/AAtZGnP0j7FjDxrfglsMpv7GTDAspNQVks5rQQsugaJpZM4KyS_p .

[SemanticBeeng]

"just use whatever the underlying client does" - of course, understood, just saying that people who want to use elastic4s may be held back by this.

Please show the way to supporting the REST client. :-)

[sksamuel]

If lots of different people each picked up a single query, would be completed in no time.

[sksamuel]

Elasticsearch HTTP client #705

[SemanticBeeng]

Indeed, but mimicking is easier than breaking new ground. Thank you for making the plan above - does not look too bad. Will keep an eye for examples you start.

And when the REST client is adopted how will the above problem go away - just drop the transport client (and, hence the netty dependency) or will you move transport client to an optional module?

[sksamuel]

elastic-core will be the common interface (client) with two implementations: elastic-tcp and elastic-http. Eventually elastic-tcp will be deleted (once Elastic themselves no longer support a TCP client - maybe never). And the reason it'll work with play, akka and spark is because the HTTP client will have few dependencies.

[SemanticBeeng]

At that time it will be interesting to evaluate elastic4s in this context: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/reference.html

[sksamuel]

What do you mean?

[SemanticBeeng]

Elasticsearch for Apache Hadoop is a project combining ElasticSearch with Spark. I am curious how using elastic4s would improve it. For example the reactive streams implementation in elastic4s might fit nicely.

[sksamuel]

I think its more just a compatibility layer. The advantage of Spark is the parallelization of tasks taking advantage of the locality of data in HDFS. I wonder, does the ES support create RDDs partitioned by the location of each shard. If not, what is it giving you other than a unified API?

[SemanticBeeng]

Using Spark and Elasticsearch for Real-time Data Analysis- Costin Leau (Elasticsearch) :https://youtu.be/afy4PkSJuzk?t=908 : "partition to partition architecture"

[sksamuel]

I'll have a watch later and see what cleverness it does :)

[danielmorozoff]

Wanted to reach back out - this slipped my mind so sorry for the delay.

I've been using the netty dependencies with es (getting the mentioned NoSuchMethodError), but it seems to be working fine. Wrote a handler for it and es seems to be working as expected. With multiple connections etc.

It fires on close, as seen in the provided code block, so you can envision properly handling the close on error (not very nice but should work)

@sksamuel & @SemanticBeeng +1 for the HTTP client- just a heads up, the JSON query syntax for ES changed from v2-5 for some of the critical methods, with them restructuring quite a bit.

[sksamuel]

Thanks for the heads up @danielmorozoff

[solicode]

The netty issue has bitten me too. For now I've created a temporary, light wrapper around the new Java REST Client (which doesn't do too much to be honest, but at least it's a starting point).

But I'd rather not turn it into yet another ES client if I continue down this path. Since the elastic4s HTTP support seems to be on its way, it might make more sense for me to help with #705 instead. Like you said, if everybody takes a query, this could be completed fairly quickly.

I do have one question though. Is the HTTP support only planned for 5.0+? Once HTTP is supported, using Amazon's Elasticsearch Service becomes an option again (since it only works with HTTP, not TCP). But Amazon ES doesn't support any versions above 2.3 right now.

It would be extra work to add the support for older versions, but if there's interest in it (or if it was planned from the start), I wouldn't mind trying to backport it from the 5.0 branch. Or however it's best to tackle it... I'm not sure yet. I think there were some elastic4s-specific refactoring done with 5.0 which might make backporting a little more complicated.

Or rather than backport, another option could be to use the elastic4s 5.0 codebase and add the older ES version support into that somehow. Like modularizing that portion of it? I don't know, that might be a bit much.

The thing is, even without doing anything for <= 2.4, I think a good chunk of the 5.0 client will end up working just fine if you point it to an older version. The low-level Java REST client version number has no relation with the ES version. The work is taking the breaking JSON changes and separating them out somehow by version.

[danielmorozoff]

@solicode we host our own es5 indices on aws without relying on the aws micro service.

Actually we had to rewrite our own low java queries as well to support the new syntax as there were changes from v 2.4

[sksamuel]

@solicode I think for now the http support will only be 5.0. It makes sense to focus on the current version first - although there's nothing stopping having both in the client. In fact, it would be easy to do since you could just write another implementation of the Executable typeclasses for the ones that differ. It'll depend on how much interest there is in helping me implement it all really.

[solicode]

@sksamuel Thanks, that sounds good. Yeah, focusing on getting the 5.0 version working first makes sense. As for the older version support after that, I should be able to help out with that too. I could bring in some help from my coworkers if it's turning out to be too much work for one person. Let me familiarize myself with the codebase a bit first, and I'll see where I can start (or if you have any thoughts like, "This is a task I should take. If somebody else wants to help they should start here" and so on, that works for me too). I'll move any future questions/comments to #705. I can probably start sometime late this week or early next week.

[SemanticBeeng]

@sksamuel another question in the planning sense for the implementation of this new design that

• isolates and makes optional tcp module to no longer stop Play! from starting
• introduces the http client (REST)

How long would it take you alone to release a first/most basic/POC version of this so we can verify?

Will you publish somewhere this POC version to work with it?

How long would it take you alone to finish a development grade version (asking to get a sense of time and difficulty)

[sksamuel]

@SemanticBeeng @solicode discussion continues on #705

[joantune]

Guys, if you are implementing REST support in elastic4s, why not do the JEST approach of adding a dependency to ES so that you can have programatic queries? I'm sure you can use those Java classes in Scala, right?

https://github.com/searchbox-io/Jest/tree/master/jest#elasticsearch-optional-dependency

On Tue, Jan 10, 2017 at 1:54 PM Stephen Samuel notifications@github.com wrote:

@SemanticBeeng https://github.com/SemanticBeeng @solicode https://github.com/solicode discussion continues on #705 https://github.com/sksamuel/elastic4s/issues/705

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sksamuel/elastic4s/issues/666#issuecomment-271580780, or mute the thread https://github.com/notifications/unsubscribe-auth/AA-DBOf2sXWY9rMidSqjHrjSc5HO326oks5rQ42egaJpZM4KyS_p .