jprante / elasticsearch-knapsack

Knapsack plugin is an import/export tool for Elasticsearch
Apache License 2.0
472 stars 77 forks source link

exported files are tiny, and compressed improperly #33

Closed zck closed 10 years ago

zck commented 10 years ago

I'm following the examples in the tutorial to export data:

zck:~$ curl -XDELETE localhost:9200/test
{"ok":true,"acknowledged":true}
zck:~$ curl -XPUT localhost:9200/test/test/1 -d '{"key":"value 1"}'
{"ok":true,"_index":"test","_type":"test","_id":"1","_version":1}
zck:~$ curl -XPUT localhost:9200/test/test/2 -d '{"key":"value 2"}'
{"ok":true,"_index":"test","_type":"test","_id":"2","_version":1}
zck:~$ curl -XPOST localhost:9200/test/test/_export?target=/log/elasticsearch.tar.gz
{"ok":true}
zck:~$ 

This is fine, but let me now look at the exported file:

zck:~$ ll /log/elasticsearch.tar.gz 
-rw-r--r-- 1 elasticsearch elasticsearch 10 Dec  4 23:58 /log/elasticsearch.tar.gz
zck:~$ tar -xf /log/elasticsearch.tar.gz 

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

While I'm at it, let me try the other export compression types.

zck:~$ curl -XPOST localhost:9200/test/test/_export?target=/log/elasticsearch.tar.bz2
{"ok":true}
zck:~$ ll /log/elasticsearch.tar.bz2 
-rw-r--r-- 1 elasticsearch elasticsearch 0 Dec  5 00:02 /log/elasticsearch.tar.bz2
zck:~$ tar -xf /log/elasticsearch.tar.bz2 
tar: This does not look like a tar archive

bzip2: Compressed file ends unexpectedly;
    perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
    Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

tar: Child returned status 2
tar: Error is not recoverable: exiting now

zck:~$ curl -XPOST localhost:9200/test/test/_export?target=/log/elasticsearch.tar.xz
{"ok":true}
zck:~$ ll /log/elasticsearch.tar.xz 
-rw-r--r-- 1 elasticsearch elasticsearch 12 Dec  5 00:03 /log/elasticsearch.tar.xz
zck:~$ tar -xf /log/elasticsearch.tar.xz 
xz: (stdin): Unexpected end of input
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Not only are none of the files right, but they're all only a few bytes.

Here's what seems to be the relevant part of the elasticsearch log file for the first .tar.gz export file:

2013-12-04 23:58:18,876][INFO ][org.xbib.elasticsearch.action.RestExportAction] [ES12] starting export to /log/elasticsearch.tar.gz
[2013-12-04 23:58:18,877][DEBUG][cluster.service          ] [ES12] processing [cluster_update_settings]: execute
[2013-12-04 23:58:18,877][DEBUG][cluster.service          ] [ES12] cluster state updated, version [108], source [cluster_update_settings]
[2013-12-04 23:58:18,878][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: execute
[2013-12-04 23:58:18,878][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: no change in cluster_state
[2013-12-04 23:58:18,879][DEBUG][cluster.service          ] [ES12] processing [cluster_update_settings]: done applying updated cluster_state
[2013-12-04 23:58:18,879][DEBUG][cluster.service          ] [ES12] processing [reroute_after_cluster_update_settings]: execute
[2013-12-04 23:58:18,881][DEBUG][cluster.service          ] [ES12] cluster state updated, version [109], source [reroute_after_cluster_update_settings]
[2013-12-04 23:58:18,881][INFO ][org.xbib.elasticsearch.action.RestExportAction] [ES12] getting settings for index test
[2013-12-04 23:58:18,881][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: execute
[2013-12-04 23:58:18,881][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: no change in cluster_state
[2013-12-04 23:58:18,882][DEBUG][cluster.service          ] [ES12] processing [reroute_after_cluster_update_settings]: done applying updated cluster_state
[2013-12-04 23:58:18,882][DEBUG][cluster.service          ] [ES12] processing [cluster_update_settings]: execute
[2013-12-04 23:58:18,882][DEBUG][cluster.service          ] [ES12] cluster state updated, version [110], source [cluster_update_settings]
[2013-12-04 23:58:18,883][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: execute
[2013-12-04 23:58:18,883][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: no change in cluster_state
[2013-12-04 23:58:18,883][DEBUG][cluster.service          ] [ES12] processing [cluster_update_settings]: done applying updated cluster_state
[2013-12-04 23:58:18,884][DEBUG][cluster.service          ] [ES12] processing [reroute_after_cluster_update_settings]: execute
[2013-12-04 23:58:18,885][DEBUG][cluster.service          ] [ES12] cluster state updated, version [111], source [reroute_after_cluster_update_settings]
[2013-12-04 23:58:18,886][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: execute
[2013-12-04 23:58:18,886][DEBUG][river.cluster            ] [ES12] processing [reroute_rivers_node_changed]: no change in cluster_state
[2013-12-04 23:58:18,887][DEBUG][cluster.service          ] [ES12] processing [reroute_after_cluster_update_settings]: done applying updated cluster_state
jprante commented 10 years ago

This is not a problem of the Knapsack plugin, but a permission problem.

Please check if you have specified the correct path for Knapsack with the path parameter.