skydive-project / skydive

An open source real-time network topology and protocols analyzer
https://skydive.network
Apache License 2.0
2.68k stars 404 forks source link

elasticsearch: dynamic_templates not applied when rolling #2347

Closed adrianlzt closed 3 years ago

adrianlzt commented 3 years ago

Skydive creates the mapping for indexes _skydive_topologylive and _skydive_topologyarchive correctly in the first boot. But when _skydive_topologyarchive rolls, the new index (_skydive_topology_archivev13-000002) does not inherit that mapping.

This new index does not flatten values set in exclude_from_mapping, producing a mapping explosion and the error:

Failed to insert entry 316: &{illegal_argument_exception Limit of total fields [1000] in index [skydive_topology_archive_v13-000002] has been exceeded     false map[] [] [] []   <nil>}
adrianlzt commented 3 years ago

roll -> RolloverIndex -> NewIndicesRolloverService.Alias -> Do()

Looks like a mappnig could be passes to the rollover API call: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html

Implemented here

safchain commented 3 years ago

@adrianlzt thanks for reporting this. Do you plan to submit a fix or do you want us to provide it ?

adrianlzt commented 3 years ago

Working on it :)

safchain commented 3 years ago

Thanks for the fix :)