Support for Elasticsearch 5.0

[Idnan]

Hi @jprante , Whats for plan for ElasticSearch 5.0?

[jprante]

I will start work soon, ES 5 has changed in many ways. Investigation needed.

[aih]

Is there any 'official' JDBC importer for ES5? If not, the ES team should really consider supporting or integrating this library.

[jprante]

The Elastic Team is providing Logstash for that purpose.

[aih]

Thanks for the quick response. I recall the native ES solutions did not work for my purpose (importing from SQL Server on Windows under Elasticsearch 1.7). Perhaps it didn't handle the mapping of my data as well as this library. Do you have a list of differences between features and support in this library vs. Logstash? Thanks again for developing and maintaining this resource.

[jprante]

I have only a rough idea about Logstash JDBC and how it works, it seems to use the JRuby features for JDBC. Not sure how Ruby interferes in such a process when SQL database types are mapped by JDBC types, then mapped to Java, then to Ruby, then to JSON, then indexed in ES and then back to an application, but there might be shortcomings. SQL to JDBC to Java is already a challenge on its own.

The JDBC importer (formerly JDBC river) was started by myself as a pure Java solution even before Logstash became a member of the Elastic company and was extended to process data via JDBC.

There is no solid plan yet to get things going, but I intend to move the JDBC importer forward with lightweight client API dependencies on Elasticsearch 5. It will take some time to find my path through the new opportunities (HTTP ES client, ingest node, task API, strict settings/mappings management, security manager). I have to find out if I can implement a JDBC plugin (again) for easier setup. That means I have to check if JDBC driver inclusion works well and easy enough under Java Security Manager. Maybe it is possible to implement a similar approach like the ingest nodes, so "JDBC nodes" could be dedicated in an ES cluster to execute JDBC only.

And beside ES 5, there are some new features waiting, mostly from the open issues here.

[aih]

The Ruby dependency alone would make me want to avoid Logstash on Windows; as you describe the multiple translations steps (SQL -> Ruby ->JDBC) suggest many possible barriers to successfully importing data.

[adamlepkowski]

What is the status of this issue?

[SteveWang123]

any progress for "es-jdbc 5.x"? Thank you in advance!

[aih]

Following @jprante's notes above, I used logstash to import from MS SQL Server to ES5. It did require installing Ruby, but it worked, even with a rather complex analysis mapping, which I applied through a logstash template.

[sandesh2014]

@jprante I know that support for ES 5.x is in progress. But out of curiosity just wanted to know if there is any expected timeline for feeder to be released for 5.x, as we are using the feeder in prod environments. We are just waiting for your release to upgrade ES to 5.x.

[apatrida]

@jprante has done a more thorough job here, but if you need ES 5 support in the meantime you can try: https://github.com/kohesive/elasticsearch-data-import-handler but don't expect the same feature set, for example CRON would be done outside the process. @jprante I hope you don't mind me putting this here, these issues are just aging so giving an alternative. I respect your work, not trying to steal from it :-D

[janhoy]

@jprante Guess many of us are waiting for a 5.x compatible release. Any status to report or a description of what's left? Possible to split the effort into smaller parts that other contributors could help with?

[aymenoss]

which jdbc importer version is compatible with Elasticsearch 5.x version ?????