Using Elasticsearch REST API

[essiembre]

From @niels, in https://github.com/Norconex/collector-http/issues/200#issuecomment-168659138:

A couple of weeks ago, I also modified the elasticsearch committer to use ES's REST interface – so I am anyway trying to get more involved in development of the ecosystem. I developed this as a separate committer so that users can choose for themselves whether to use the Node client (of the existing committer) or the REST client (of my new committer).

As I haven't yet gotten around to writing a test suite, I was thus far too hesitant to "officially submit" the new committer to you. But seeing how I am critiquing your "secret" commits, it is only fair to expose my own.. So here it is, if you are interested in taking a peek: https://github.com/herimedia/norconex-committer-elasticsearch-rest.

[essiembre]

@niels, after checking with @pascaldimassimo, the latest (snapshot) version of Elasticsearch Committer now gives the option to use the Transport Client instead of the Node Client. That's not the same as using the REST API, but it is apparently the recommended usage when using the Node Client is not possible.

Because the option to use the Transport Client is now there, and it is more efficient than using the REST API (when using a Java client at least), we do not see advantages to also support the REST API. Do you?

In any case, I am quite grateful for your contribution and I decided to provide a link to it in a new Contributions page I just created

[niels]

Thanks @essiembre!

The transport client is a good option to have but does not address the common use case of putting elasticsearch behind a HTTP proxy as it uses the proprietary elasticsearch transport protocol (rather than HTTP).

Putting ES behind an HTTP proxy is a popular way of i.a. adding encryption (HTTPS) and authentication (HTTP Basic Auth).

[danizen]

Agreed. I vote for basic capability of using the REST client.

[pascaldimassimo]

For the latest versions, Elasticsearch announced that the java and transport APIs will soon be deprecated in favour of the REST client. So it might be necessary at some point to promote the client contributed by @niels

[essiembre]

Hi @pascaldimassimo! A REST version is now in the work to replace the transport one, taking inspiration from @niels and @danizen. It should be available in the very near future. Thanks to all for your input.

[danizen]

Okay - I urge you to include HTTP basic authentication, similar code to what you do for Mongo, but just basic auth.

[danizen]

@essiembre - note https://github.com/herimedia/norconex-committer-elasticsearch-rest/pull/4

[essiembre]

Done! Try the new snapshot release.

For usage, refer to Javadoc documentation for now until the site gets updated.

Given Elasticsearch decided to eventually stop supporting the transport client, I replaced it with the REST API. Those wanting to keep using the transport client will have to stick to version 3.

@danizen It does support basic authentication, but that's the only thing I did not get a chance to test.

Please give it a try and confirm.

[danizen]

I wanted to give feedback that this is working for me. I've removed my custom code and moved to the snapshot releases of collector-core, collector-http, and committer-elasticsearch. My project now contains no Java, which I will have to fix soon.

[essiembre]

Can you confirm the authentication part as well?

[sveba]

Hi, first of all, thanx for the great job. When can we expect a release for this? Version 4 with REST support I mean. I can access only the official MVN repo and even the SNAPSHOT can not be found there. Thanx

[essiembre]

Ask and you shall receive! I was waiting for @danizen to confirm the authentication part is working fine for him, but given the time, I am pretty sure it is. You can get the 4.0.0 release here. If you run into issues, please create a new ticket.

[sveba]

Great, hoping to see it soon in the mvn official repo too. thx

[essiembre]

It is already deployed here: https://oss.sonatype.org/content/repositories/releases/com/norconex/collectors/norconex-committer-elasticsearch/4.0.0/

It may take up to a few hours for it gets propagated to Maven Central.

[sveba]

You are right. It takes time.