search

jprante / elasticsearch-knapsack

Knapsack plugin is an import/export tool for Elasticsearch
Apache License 2.0
472 stars 77 forks source link

_import does not work in v2.3.4 #119

Open Maddy7868 opened 7 years ago

Maddy7868 commented 7 years ago

Hi Jörg,

I have ElasticSearch v.0.20.6 with plugin 0.20.6.2 and have another instance on same machine v2.3.4 plugin v2.3.4.0. I ran the following commands as mentioned in the release notes.

Please help me out in this.

ElasticSearch v0.20.6

curl -XDELETE localhost:9200/test curl -XPUT 'localhost:9200/test/test/1' -d '{"key":"value 1"}' curl -XPUT 'localhost:9200/test/test/2' -d '{"key":"value 2"}' curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}' curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}' curl -XPOST 'localhost:9200/test/_export'

CYGWIN LOG

XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XDELETE localhost:9200/test {"ok":true,"acknowledged":true} XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XPUT localhost:9200/test/test/1 -d '{"key":"value 1"}' {"ok":true,"_index":"test","_type":"test","_id":"1","_version":1} XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XPUT localhost:9200/test/test/2 -d '{"key":"value 2"}' {"ok":true,"_index":"test","_type":"test","_id":"2","_version":1} XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}' {"ok":true,"_index":"test2","_type":"foo","_id":"1","_version":2} XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}' {"ok":true,"_index":"test2","_type":"bar","_id":"1","_version":2} XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XPOST 'localhost:9200/test/_export' {"running":true,"state":{"mode":"export","started":"2017-05-18T19:26:41.001Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Agon"}}

COPIED the tar.gz to C:\Users\XXXX\Personal\elasticsearch-2.3.4\logs

STOPPED ELASTICSEARCH v0.20.6 STARTED ELASTICSEARCH v2.3.4

ElasticSearch v2.3.4

curl -XDELETE 'localhost:9200/test' curl -XPOST 'localhost:9200/test/_import curl -XPOST 'localhost:9200/test/_search?q=*&pretty'

CYGWIN console

XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XPOST 'localhost:9200/test/_import' {"running":true,"state":{"mode":"import","started":"2017-05-18T19:33:37.576Z","p ath":"file:///C:/Users/XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Spider Doppelganger"}} XXXX@PBVWWSU1 /cygdrive/c/Users/XXXX/Downloads $ curl -XPOST 'localhost:9200/test/_search?q=*&pretty' { "error" : { "root_cause" : [ { "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_or_alias", "resource.id" : "test", "index" : "test" } ], "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_or_alias", "resource.id" : "test", "index" : "test" }, "status" : 404 }

Elastic Search Console

[2017-05-18 14:31:34,367][INFO ][node ] [Spider Doppelganger ] started [2017-05-18 14:31:34,704][INFO ][cluster.routing.allocation] [Spider Doppelgange r] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.knapsack][4], [.knapsack][2], [.knapsack][2], [.knapsack][4]] ...]). [2017-05-18 14:32:04,077][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:32:34,091][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:33:04,104][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:33:34,115][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:33:37,578][INFO ][KnapsackImportAction ] start of import: {"m ode":"import","started":"2017-05-18T19:33:37.576Z","path":"file:///C:/Users/ XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Spider Dopp elganger"} [2017-05-18 14:33:37,791][INFO ][KnapsackImportAction ] end of import: {"mod e":"import","started":"2017-05-18T19:33:37.576Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Spider Doppel ganger"}, count = 4 [2017-05-18 14:34:04,128][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:34:34,142][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:35:04,153][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:35:34,165][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node [2017-05-18 14:36:04,179][INFO ][cluster.routing.allocation.decider] [Spider Dop pelganger] low disk watermark [85%] exceeded on [ep-3aQs6TU6CjXCxQmFP-A][Spider Doppelganger][C:\Users\XXXX\Personal\elasticsearch-2.3.4\data\elasticsea rch\nodes\0] free: 8.7gb[14.6%], replicas will not be assigned to this node

Thanks, Maddy

jprante commented 7 years ago

Please check low disk watermark [85%] exceeded error message, it means that not much data can be imported because of precautions in Elasticsearch 2.3.4

Maddy7868 commented 7 years ago

Thanks for you reply. I don't see the low disk watermark error messages now. But I still don't see the data in the import. Please could you help me to identify the issue.

ES 0.20.6

Command

curl -XDELETE localhost:9200/test curl -XPUT 'localhost:9200/test/test/1' -d '{"key":"value 1"}' curl -XPUT 'localhost:9200/test/test/2' -d '{"key":"value 2"}' curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}' curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}' curl -XPOST 'localhost:9200/test/_export' tar zxvf test.tar.gz test/_settings echo '{"index.number_of_shards":"1","index.number_of_replicas":"0"}' > test/_settings/null/null

Log

[2017-05-22 12:07:02,040][INFO ][node ] [Emma Frost] {0.20.6}[9360]: initializing ... [2017-05-22 12:07:02,071][INFO ][plugins ] [Emma Frost] loaded[knapsack-0.20.6.2-${build], sites []

[2017-05-22 12:07:05,884][INFO ][node ] [Emma Frost] {0.20.6}[9360]: starting ... [2017-05-22 12:07:06,098][INFO ][transport ] [Emma Frost] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.230.34.132:9300] } [2017-05-22 12:07:09,384][INFO ][cluster.service ] [Emma Frost] new_master [Emma Frost][VMfT-GqOQ9yoyrPpPEji-Q][inet[/10.230.34.132:9300]], reason: zen -disco-join (elected_as_master) [2017-05-22 12:07:09,416][INFO ][discovery ] [Emma Frost] elasticsearch/VMfT-GqOQ9yoyrPpPEji-Q [2017-05-22 12:07:09,499][INFO ][http ] [Emma Frost] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.230.34.132:9200] }

[2017-05-22 12:07:09,515][INFO ][gateway ] [Emma Frost] recovered [0] indices into cluster_state [2017-05-22 12:08:54,038][INFO ][cluster.metadata ] [Emma Frost] [test]creating index, cause [auto(index api)], shards [5]/[1], mappings [] [2017-05-22 12:08:54,869][INFO ][cluster.metadata ] [Emma Frost] [test]update_mapping [test] (dynamic) [2017-05-22 12:08:55,041][INFO ][cluster.metadata ] [Emma Frost] [test2] creating index, cause [auto(index api)], shards [5]/[1], mappings [] [2017-05-22 12:08:55,327][INFO ][cluster.metadata ] [Emma Frost] [test2] update_mapping [foo] (dynamic) [2017-05-22 12:08:55,382][INFO ][cluster.metadata ] [Emma Frost] [test2] update_mapping [bar] (dynamic) [2017-05-22 12:08:59,716][INFO ][KnapsackService ] add: plugin.knapsack.export.state -> [] [2017-05-22 12:08:59,722][INFO ][KnapsackExportAction ] start of export: {"mode":"export","started":"2017-05-22T17:08:59.669Z","path":"file:///C:/Users/ XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Emma Frost" } [2017-05-22 12:08:59,722][INFO ][KnapsackService ] update cluster settings: plugin.knapsack.export.state -> [{"mode":"export","started":"2017-05-22T17: 08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Emma Frost"}] [2017-05-22 12:08:59,723][INFO ][KnapsackExportAction ] getting settings for indices [test] [2017-05-22 12:08:59,726][INFO ][KnapsackExportAction ] found indices: [test] [2017-05-22 12:08:59,747][INFO ][KnapsackExportAction ] getting mappings for index test and types [] [2017-05-22 12:08:59,760][INFO ][KnapsackExportAction ] found mappings: [test] [2017-05-22 12:08:59,765][INFO ][KnapsackExportAction ] adding mapping: test [2017-05-22 12:08:59,766][INFO ][KnapsackExportAction ] getting aliases forindex test [2017-05-22 12:08:59,767][INFO ][KnapsackExportAction ] found 0 aliases [2017-05-22 12:08:59,872][INFO ][KnapsackExportAction ] end of export: {"mode":"export","started":"2017-05-22T17:08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz","node_name":"Emma Frost"}, packets = 4, total bytes transferred = 177, rate = 35.400000 [2017-05-22 12:08:59,891][INFO ][KnapsackService ] remove: plugin.knapsack.export.state -> [{"mode":"export","started":"2017-05-22T17:08:59.669Z","path ":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/bin/test.tar.gz", "node_name":"Emma Frost"}] [2017-05-22 12:08:59,893][INFO ][KnapsackService ] update cluster settings: plugin.knapsack.export.state -> [{"mode":"export","started":"2017-05-22T17: 08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch-0.20.6/ bin/test.tar.gz","node_name":"Emma Frost"}]

ES 0.20.6 cygwin logs XXXX@PBVWWSU1 $ curl -XDELETE localhost:9200/test {"error":"IndexMissingException[[test] missing]","status":404} XXXX@PBVWWSU1 $ curl -XPUT 'localhost:9200/test/test/1' -d '{"key":"value 1"}' curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}' curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}' curl -XPOST 'localhost:9200/test/_export'{"ok":true,"_index":"test","_type":"test","_id":"1","_version":1} XXXX@PBVWWSU1 $ curl -XPUT 'localhost:9200/test/test/2' -d '{"key":"value 2"}' {"ok":true,"_index":"test","_type":"test","_id":"2","_version":1} XXXX@PBVWWSU1 $ curl -XPUT 'localhost:9200/test2/foo/1' -d '{"key":"value 1"}' {"ok":true,"_index":"test2","_type":"foo","_id":"1","_version":1} XXXX@PBVWWSU1 $ curl -XPUT 'localhost:9200/test2/bar/1' -d '{"key":"value 1"}' {"ok":true,"_index":"test2","_type":"bar","_id":"1","_version":1} XXXX@PBVWWSU1 $ curl -XPOST 'localhost:9200/test/_export' {"running":true,"state":{"mode":"export","started":"2017-05-22T17:08:59.669Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch- 0.20.6/bin/test.tar.gz","node_name":"Emma Frost"}} XXXX@PBVWWSU1 $ tar zxvf test.tar.gz test/_settings test/_settings/null/null

XXXX@PBVWWSU1 $ echo '{"index.number_of_shards":"1","index.number_of_replicas":"0"}' > test/_settings/null/null

STOPPED ES 0.20.6 STARTED ES 2.3.4 COPIED THE test.tar.gz under C:\Users\XXXX\Personal\elasticsearch-2.3.4\logs directory

ES 2.3.4 ++++++++++++++++++++++++++++++++++++++

Commands

curl -XDELETE 'localhost:9200/test' curl -XPOST 'localhost:9200/test/_import?test_settings=test/_settings/null/null' curl -XGET 'localhost:9200/test/_settings?pretty' curl -XPOST 'localhost:9200/test/_search?q=*&pretty'

C:\Users\XXXX\Personal\elasticsearch-2.3.4\bin>elasticsearch.bat [2017-05-22 12:15:20,823][WARN ][bootstrap ] jvm uses the clientvm, make sure to run java with the server vm for best performance by adding - server to the command line [2017-05-22 12:15:21,321][INFO ][node ] [Razor Fist] version[2.3.4], pid[8604], build[e455fd0/2016-06-30T11:24:31Z] [2017-05-22 12:15:21,322][INFO ][node ] [Razor Fist] initializing ... [2017-05-22 12:15:22,355][INFO ][plugins ] [Razor Fist] modules [reindex, lang-expression, lang-groovy], plugins [knapsack], sites [] [2017-05-22 12:15:22,425][INFO ][env ] [Razor Fist] using [1] data paths, mounts [[SYSTEM (C:)]], net usable_space [9.6gb], net total_space [59.6gb], spins? [unknown], types [NTFS] [2017-05-22 12:15:22,425][INFO ][env ] [Razor Fist] heap size [989.8mb], compressed ordinary object pointers [unknown] [2017-05-22 12:15:25,988][INFO ][node ] [Razor Fist] initialized [2017-05-22 12:15:25,988][INFO ][node ] [Razor Fist] starting ...

[2017-05-22 12:15:26,297][INFO ][discovery ] [Razor Fist] elasticsearch/Jm_HLWQbQpGl9tpLTFn1fg [2017-05-22 12:15:30,416][INFO ][cluster.service ] [Razor Fist] new_master {Razor Fist}{Jm_HLWQbQpGl9tpLTFn1fg}{127.0.0.1}{127.0.0.1:9300}, reason: zen -disco-join(elected_as_master, [0] joins received) [2017-05-22 12:15:30,462][INFO ][gateway ] [Razor Fist] recovered [0] indices into cluster_state

[2017-05-22 12:15:30,551][INFO ][node ] [Razor Fist] started [2017-05-22 12:17:55,383][INFO ][KnapsackImportAction ] start of import: {"mode":"import","started":"2017-05-22T17:17:55.380Z","path":"file:///C:/Users/ XXXX/Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Razor Fist"} [2017-05-22 12:17:55,560][INFO ][cluster.metadata ] [Razor Fist] [.knapsack] creating index, cause [api], templates [], shards [5]/[1], mappings [] [2017-05-22 12:17:55,969][INFO ][cluster.routing.allocation] [Razor Fist] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.knaps ack][4]] ...]). [2017-05-22 12:18:25,878][INFO ][cluster.metadata ] [Razor Fist] [.knapsack] create_mapping [knapsack] [2017-05-22 12:18:26,087][INFO ][KnapsackImportAction ] end of import: {"mode":"import","started":"2017-05-22T17:17:55.380Z","path":"file:///C:/Users/XXXX /Personal/elasticsearch-2.3.4/logs/test.tar.gz","node_name":"Razor Fist"}, count = 4

Cygwin Logs +++++++++++++++++++++++++++++++++++++++++ XXXX@PBVWWSU1 $ curl -XDELETE 'localhost:9200/test' {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"test", "index":"test"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"test","index": "test"},"status":404} XXXX@PBVWWSU1 $ curl -XPOST 'localhost:9200/test/_import?test_settings=test/_settings/null/null' {"running":true,"state":{"mode":"import","started":"2017-05-22T17:17:55.380Z","path":"file:///C:/Users/XXXX/Personal/elasticsearch- 2.3.4/logs/test.tar.gz","node_name":"Razor Fist"}} XXXX@PBVWWSU1 $ curl -XGET 'localhost:9200/test/_settings?pretty' { "error" : { "root_cause" : [ { "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_or_alias", "resource.id" : "test", "index" : "test" } ], "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_or_alias", "resource.id" : "test", "index" : "test" }, "status" : 404 }

XXXX@PBVWWSU1 $ curl -XPOST 'localhost:9200/test/_search?q=*&pretty' { "error" : { "root_cause" : [ { "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_or_alias", "resource.id" : "test", "index" : "test" } ], "type" : "index_not_found_exception", "reason" : "no such index", "resource.type" : "index_or_alias", "resource.id" : "test", "index" : "test" }, "status" : 404 }