search

ncolomer / elasticsearch-osmosis-plugin

An Osmosis plugin that index OpenStreetMap data into elasticsearch
Apache License 2.0
101 stars 34 forks source link

ElasticSearch 5.x.x support #27

Open dovka opened 7 years ago

dovka commented 7 years ago

I installed the osmosis and the plugin per instructions, but it looks like ElasticSearch version 5.x.x is not supported?

ElasticSearch cluster Is up:

curl http://localhost:9200/_cluster/health\?pretty\=true { "cluster_name" : "elasticsearch", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 16, "active_shards" : 16, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 16, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 50.0 }

Osmosis error:

11:59:51.842 [main] DEBUG org.elasticsearch.client.transport - [Tag] adding address [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}] 11:59:52.016 [elasticsearch[Tag][management][T#1]] DEBUG org.elasticsearch.transport.netty - [Tag] connected to node [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}] 11:59:52.037 [elasticsearch[Tag][generic][T#1]] DEBUG org.elasticsearch.transport.netty - [Tag] disconnecting from [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}], channel closed event 11:59:52.104 [elasticsearch[Tag][generic][T#2]] INFO org.elasticsearch.client.transport - [Tag] failed to get local cluster state for {#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}, disconnecting... org.elasticsearch.transport.NodeDisconnectedException: [][localhost/127.0.0.1:9300][cluster:monitor/state] disconnected May 21, 2017 11:59:52 AM org.openstreetmap.osmosis.core.Osmosis main SEVERE: Execution aborted. NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207) at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347) at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:841) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59) at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.checkConnection(ElasticsearchClientBuilder.java:116) at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.build(ElasticsearchClientBuilder.java:60) at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.buildElasticsearchClient(ElasticSearchWriterFactory.java:83) at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.createTaskManagerImpl(ElasticSearchWriterFactory.java:29) at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:60) at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50) at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112) at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86) at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

ElasticSearch Error: [2017-05-19T16:07:01,871][INFO ][o.e.c.r.a.DiskThresholdMonitor] [FvT7WdR] rerouting shards: [high disk watermark exceeded on one or more nodes] [2017-05-19T16:07:06,969][WARN ][o.e.t.n.Netty4Transport ] [FvT7WdR] exception caught on transport layer [[id: 0x240b7a45, L:/172.18.0.3:9300 - R:/172.18.0.1:41884]], closing connection java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1354) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[transport-netty4-5.4.0.jar:5.4.0] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.9.Final.jar:4.1.9.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.9.Final.jar:4.1.9.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.9.Final.jar:4.1.9.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:524) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:478) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) [netty-transport-4.1.9.Final.jar:4.1.9.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.9.Final.jar:4.1.9.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

jpierre03 commented 7 years ago

Hi,

Le Sun, 21 May 2017 02:05:27 -0700, David notifications@github.com a écrit :

ElasticSearch Error: [2017-05-19T16:07:01,871][INFO ][o.e.c.r.a.DiskThresholdMonitor] [FvT7WdR] rerouting shards: [high disk watermark exceeded on one or more nodes] [2017-05-19T16:07:06,969][WARN ][o.e.t.n.Netty4Transport ] [FvT7WdR] exception caught on transport layer [[id: 0x240b7a45, L:/172.18.0.3:9300 - R:/172.18.0.1:41884]], closing connection java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0]

It seems there is more than one error in you ES cluster:

" ElasticSearch Error: [2017-05-19T16:07:01,871][INFO ][o.e.c.r.a.DiskThresholdMonitor] [FvT7WdR] rerouting shards: [high disk watermark exceeded on one or more nodes] "

-> At least one storage is almost full.

" [2017-05-19T16:07:06,969][WARN ][o.e.t.n.Netty4Transport ] [FvT7WdR] exception caught on transport layer [[id: 0x240b7a45, L:/172.18.0.3:9300 - R:/172.18.0.1:41884]], closing connection java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0] "

-> One Elasticsearh instance in cluster in from 2.x version.

-- Jean-Pierre

dovka commented 7 years ago

Thank you for the quick answer. 1) Storage error has been addressed, no impact on the bottom line, so it is irrelevant 2) gotten that error " unsupported version" while running on single node cluster (localhost) with version 5.2.2 and 5.4.0 (I mean I tried sequentially with ElasticSearch 5.2.2 and when upgraded to 5.4.0 and tried again wit the same result )

it seems to me that error "version: [2.0.0] minimal compatible version is: [5.0.0]" is about the CLIENT that is using protocols for version 2.0.0?

jpierre03 commented 7 years ago

Indeed according to https://github.com/ncolomer/elasticsearch-osmosis-plugin compatibility table, latest release is compatible with Elasticsearch 2.x

dovka commented 7 years ago

2.x.x.increasingly becoming a history, if this project is to stay alive, needs 5.x.x support :)

BTW, when I downgraded to 2.4.5, I run into issue #26, so It again failed to work :(

ncolomer commented 7 years ago

@dovka indeed ES 5.x branch is not supported yet. Actually, I have no plan to implement it by myself (time missing), but any PR is welcome! Cheers, Nicolas PS: thank you @jpierre03 for your answers!