amazon-archives / cloudwatch-logs-subscription-consumer

A specialized Amazon Kinesis stream reader (based on the Amazon Kinesis Connector Library) that can help you deliver data from Amazon CloudWatch Logs to any other system in near real-time using a CloudWatch Logs Subscription Filter.
Other
397 stars 152 forks source link

How to map a geo_point in indices that are created automatically by elasticsearch-kopf? #14

Open muhammadsr opened 8 years ago

muhammadsr commented 8 years ago

I would really appreciate any help with the following issue:

Take a look at the mapping for the following index

{
"_default_":{
"properties":{
"@timestamp":{
"format":"dateOptionalTime",
"doc_values":true,
"type":"date"
},
"@message":{
"type":"string"
},
"@id":{
"type":"string"
}
},
"_all":{
"enabled":false
}
},
"development":{
"properties":{
"@timestamp":{
"format":"dateOptionalTime",
"doc_values":true,
"type":"date"
},
"@log_stream":{
"type":"string"
},
"@message":{
"type":"string"
},
"Context":{
"properties":{
"LocationId":{
"type":"string"
},
"SubCategoryId":{
"type":"string"
},
"HttpServerName":{
"type":"string"
},
"HttpRequestUri":{
"type":"string"
},
"CategoryId":{
"type":"string"
},
"RequestId":{
"type":"string"
},
"Coordinate":{
"type":"string"
},
"ServiceId":{
"type":"string"
},
"UserId":{
"type":"string"
},
"HttpMethod":{
"type":"string"
}
}
},
"Message":{
"type":"string"
},
"@id":{
"type":"string"
},
"Thread":{
"properties":{
"Name":{
"type":"string"
},
"Id":{
"type":"long"
},
"Priority":{
"type":"long"
}
}
},
"Timestamp":{
"format":"dateOptionalTime",
"type":"date"
},
"Marker":{
"type":"string"
},
"@log_group":{
"type":"string"
},
"@owner":{
"type":"string"
}
},
"_all":{
"enabled":false
}
}
}

From the mapping above, you can see that the Coordinate property type is a stringtype but it would be nice if I can find a way to ensure that this property is of type geo_point.

Keep in mind that if I manually change the mapping for Coordinate to geo_point, it will work and Kibana will recognize it as a geo_point type. However, when kopf automatically creates another daily index, it will map Coordinate as a string type and Kibana will get a mapping conflict.

https://cdn.discourse.org/elastic/uploads/default/optimized/2X/9/99d5ed421e580787948626382ab547ec17bbbd56_1_690x339.png

dvassallo commented 8 years ago

I think what you need is to specify your index mappings in an index template with the cwl-* prefix. There is already an existing index template that you can modify, or else you can add a new index template from the Elasticsearch API.