Import settings mappings

[thanostollos]

Hi Burnes,

I'm Thanos, a web developer http://gravatar.com/thantoldo :)

I have a question about the elasticsearch-inout-plugin.

I have exported an elasticsearch index, with the following command curl -X POST 'http://localhost:9200/index1/_export' -d '{ "fields": [ "_index", "_type", "_id", "_version", "_source", "_timestamp"], "output_file": "/root/dump/dump-${index}-${shard}", "force_overwrite": true } '

A sample entry is shown below (it is a really basic entry as you can see) {"_index":"index1","_type":"pin","_id":"GdfkapBkT9m_6qD43287uw","_version":1,"_source":{"sid":"sid123"},"_timestamp":"[0 0 1 3e 69 66 6b 14]"}

I am creating a new index (index2) with the same mapping and I am importing with the command curl -X POST 'http://localhost:9200/index2/_import' -d '{ "directory": "/root/dump" } '

The problem is that the _timestamp fields in index2 represent the time the index2 is imported and not the index1 timestamps.

I also checked to duplicate the index1 at a different elasticsearch setup as index1 and also tried curl -X POST 'http://localhost:9200/_export' and curl -X POST 'http://localhost:9200/index1/_import' with the same results.

Any hint?

[burnes]

Is the _timestamp property in the index mapping set to store: yes? The time stamp does not seem to be in correct format, is this a custom format?

[thanostollos]

The mapping is
"mappings": { "pin": { "_timestamp" : { "enabled" : true, "store" : true, "format" : "YYYY-MM-dd HH:mm:ss" }, .....

The value "0 0 1 3e 69 66 6b 14" is in hex, equals to 1367567919892 which is the value elasticsearch returns with "curl 'http://localhost:9200/index1/_search?pretty=1&fields=_source,_timestamp'". So I assumed that the timestamp is exported correctly.

Also, I forgot to mention, that the _version field has gone when importing, which is also wrong.

I suggest trying to duplicate this yourself with your _timestamp mapped data, when you are available.

Thanks :)

[burnes]

Here is what i tested (empty Elasticsearch Instance):

curl -XPUT localhost:9200/index1 -d '{"settings": {"number_of_shards":2, "number_of_replicas": 0}}'

curl -XPOST localhost:9200/index1/pin/_mapping -d '{"pin":{"_timestamp":{"enabled":true, "store": "yes", "format": "YYYY-MM-dd HH:mm:ss"}, "properties": {"sid": {"type":"string"}}}}}'

curl -XPOST localhost:9200/index1/pin/id1 -d '{"sid": "sid123"}' curl -XPOST localhost:9200/index1/pin/id1 -d '{"sid": "sid123.version2"}'

curl -X POST 'http://localhost:9200/index1/_export' -d '{"fields":["_index", "_type", "_id", "_version", "_source", "_timestamp"], "output_file": "/tmp/dump/dump-${index}-${shard}", "force_overwrite": true}'

One export file contains this: {"_index":"index1","_type":"pin","_id":"id1","_version":2,"_source":{"sid":"sid123.version2"},"_timestamp":1368521107666}

curl -XPUT localhost:9200/index2 -d '{"settings": {"number_of_shards":2, "number_of_replicas": 0}}'

curl -XPOST localhost:9200/index2/pin/_mapping -d '{"pin":{"_timestamp":{"enabled":true, "store": "yes", "format": "YYYY-MM-dd HH:mm:ss"}, "properties": {"sid": {"type":"string"}}}}}'

curl -X POST 'http://localhost:9200/index2/_import' -d '{"directory": "/tmp/dump"}'

This way the result is correct:

curl localhost:9200/index2/pin/id1?fields=_timestamp,_source

{"_index":"index2","_type":"pin","_id":"id1","_version":2,"exists":true, "_source" : {"sid":"sid123.version2"},"fields":{"_timestamp":1368521107666}}

Please retry these steps, I can not reproduce your failures, maybe there is some other problem with configuration or global settings.

[thanostollos]

I duplicated your setup correctly! When I find out what is going on, I 'll post it here. The odd is the hex _timestamp stored value..

Thanks for your support :)

[thanostollos]

Well, I recreated my main index and everything exported/imported fine. It seems it was a mapping issue with my setup..

Cheers :)