search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.2k stars 3.5k forks source link

logstash 7.0 broke template upload via manage_template (my custom index_patterns overwritten) #10687

Open Mekk opened 5 years ago

Mekk commented 5 years ago

I just upgraded to logstash 7.0 and it turned out that it no longer properly uploads my template mapping - it dropped my index_patterns replacing them with logstash-*. As template name was kept, effectively my template mapping was no longer present and some data misinterpreation and even loss resulted (due to invalid typing, loss in cases where conflicts arose)

What happens

I use custom index prefix (on purpose, this is specific installation targeted on custom logs which are to be kept separate from standard logstash-ized data). So my dfx.json contains:

{
        "index_patterns": ["dfx-*", "xdfx-*"],
       /* settings, mappings… */
}

My logstash configuration targets those indexes and also asks logstash to upload the mapping. Crucial snippet

output {
 elasticsearch {
    # … skip network args …
    index => "dfx-%{[@metadata][schema_version]}-%{+xxxx.ww}"

    manage_template => true
    template_name => "dfx"
    template_overwrite => true
    template => "/etc/logstash/mappings/dfx.json"
}

On logstash 6.6 and 6.7 it worked fine. On 7.0 it does not. Logstash happily uploads my template (which I fixed to work on ES 7.0, removing type), but it overwrites it partially. In fact, GET _template/dfx proves, that uploaded template is:

    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "logstash-policy",
          "rollover_alias" : "logstash"
        },
       /* rest of the settings as I wrote */
   },
  /* mappings as I wrote */

Note completely different index_patterns!

This is of course very bad: as logstash overwrote dfx template, my older version is no longer present, current template targets only logstash-* and my dfx-… indexes no longer have mapping template. So once new dfx-… index was created, it turned out to be template-less and felt to type deduction. As my data is complicated, this brought various type conflicts (not to mention broken visualizations where keywords turned into text). I suppose also „normal” logstash-… index wouldn't be too happy to pick this template.

I undertand the whole problem is related to now-default ilm-policy, but I suppose assuming everybody uses only indexes named logstash-* is going too far.

Workaround? I stopped using manage_template. But that's pity, it was nice to install template and logtash configuration together, now i have to coordinate logstash config installation with manual template upload…

Mekk commented 5 years ago

Better workaround: explicit

ilm_enabled => false

reverts to the old behaviour. So I did it and more-or-less resolved the problem.

Still I feel there are few things wrong:

  1. At the very least logstash should detect inconsistency between index_patterns inside mapping and index_patterns it is to rewrite file with (and probably report an error and give up uploading template). Of course there can be an option to ignore this mismatch, but by default something should happen.

  2. In case ilm_rollover_alias is not defined, defaulting it to logstash in case index is defined to sth else seems a strange and confusing idea. It would make more sense to default to whatever index is, or just report an error and require rollover alias to be set

  3. Unless I miss something, it is not possible to use %{variables} in ilm_rollover_alias (while it is possible to use them in index). So it doesn't seem possible to write this configuration properly in case index name is varying.

I am not sure whether case of non-standard indexes is worth being resolved fully (after all, with advanced config, it is probably better to write rollover settings by hand) but at the very least I think sth should be done to avoid mistakes like mine.

localhorst-org commented 5 years ago

this is the same for me,

template, index patterns name striped with elk 7.0.0

to get it clear for myself, all the following is made with new Installations in VirtualBox under Ubuntu 16.04

elk is installed with apt from 

deb https://artifacts.elastic.co/packages/6.x/apt stable main
deb https://artifacts.elastic.co/packages/7.x/apt stable main

i will discribed first my expected and working behavior from elk version 6.7.1 and earlier.

install and start elasticsearch

no index no own template

:~# curl -s 'localhost:9200/_cat/indices?v' | sort -r
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

:~# curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
{ }

create a template.json file

:~$ vim elasticsearch-template_v00005.json
{
        "index_patterns": "logstash_v00005-*",
        "settings": {
                "index" : {
                        "refresh_interval": "10s" ,
                        "number_of_shards" : 1,
                        "number_of_replicas" : 0
                }
       }
}

import template and check 

:~$ curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9200/_template/logstash_v00005?pretty' -d@elasticsearch-template_v00005.json
{
  "acknowledged" : true
}
:~# tail -f /var/log/elasticsearch/elasticsearch.log
[...]
[2019-04-16T12:11:10,450][INFO ][o.e.c.m.MetaDataIndexTemplateService] [0aJE0Em] adding template [logstash_v00005] for index patterns [logstash_v00005-*]
:~$ curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
{
  "logstash_v00005" : {
    "order" : 0,
    "index_patterns" : [
      "logstash_v00005-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "0",
        "refresh_interval" : "10s"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

install logstash and at that point only my out konfiguration

:~# cat /etc/logstash/conf.d/98-elasticsearch-output.conf
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => true
    template_overwrite => true

    template => '/etc/logstash/elasticsearch-template_v00005.json'
    template_name => 'logstash_v00005'
    index => 'logstash_v00005-%{+YYYY.MM.dd}'
  }
  stdout { codec => rubydebug }
}

start logstash

:~# tail -f /var/log/elasticsearch/elasticsearch.log
[...]
[2019-04-16T12:14:29,864][INFO ][o.e.c.m.MetaDataIndexTemplateService] [0aJE0Em] adding template [logstash_v00005] for index patterns [logstash_v00005-*]
:~# curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
{
  "logstash_v00005" : {
    "order" : 0,
    "index_patterns" : [
      "logstash_v00005-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "0",
        "refresh_interval" : "10s"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

-> template is the same and the index patterns name is expected logstash_v00005-*

now with elk 7.0.0 in a new Ubuntu 16.04 installation

install elasticsearch and start

:~$ curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
{ }

-> same template.json file like above

:~$ curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9200/_template/logstash_v00005?pretty' -d@elasticsearch-template_v00005.json
{
  "acknowledged" : true
}
:~# tail -f /var/log/elasticsearch/elasticsearch.log
[...]
[2019-04-16T10:47:38,497][INFO ][o.e.c.m.MetaDataIndexTemplateService] [devubun1604] adding template [logstash_v00005] for index patterns [logstash_v00005-*]
:~$ curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
{
  "logstash_v00005" : {
    "order" : 0,
    "index_patterns" : [
      "logstash_v00005-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "0",
        "refresh_interval" : "10s"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

-> looks good so far

install logstash. out config like above, start logstash

:~# tail -f /var/log/elasticsearch/elasticsearch.log
[...]
[2019-04-16T10:57:35,684][INFO ][o.e.c.m.MetaDataIndexTemplateService] [devubun1604] adding template [logstash_v00005] for index patterns [logstash-*]
:~$ curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
{
  "logstash_v00005" : {
    "order" : 0,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "logstash-policy",
          "rollover_alias" : "logstash"
        },
        "refresh_interval" : "10s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

-> this is looking for me my 98-elasticsearch-output.conf configuration is ignored and the elk default behavior is working and overrites my template with the not expected index patterns name logstash-*

mybe to make it clear in a short conclusion with start logstash

expected 6.7.x behavior index patterns name is logstash_v00005-*

[2019-04-16T10:47:38,497][INFO ][o.e.c.m.MetaDataIndexTemplateService] [devubun1604] adding template [logstash_v00005] for index patterns [logstash_v00005-*]

with 7.0.0 striped the name from index patterns

[2019-04-16T10:57:35,684][INFO ][o.e.c.m.MetaDataIndexTemplateService] [devubun1604] adding template [logstash_v00005] for index patterns [logstash-*]

regards horst

localhorst-org commented 5 years ago

and within the template 

:~# curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
[...]
        "number_of_replicas" : "0"
[...]

will be ignored by index creation 

:~# curl -s localhost:9200/_cat/indices?v
health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash_v00005-2019.04.18 tENJSbKtTxCBO4A2GYa2XA   1   1        117            0    273.4kb        273.4kb

and mybe the hole template will be ignored

:~# curl -XDELETE 'http://localhost:9200/logstash_v00005-2019.04.18?pretty'
{
  "acknowledged" : true
}
[2019-04-18T11:15:48,463][INFO ][o.e.c.m.MetaDataDeleteIndexService] [devubun1604] [logstash_v00005-2019.04.18/tENJSbKtTxCBO4A2GYa2XA] deleting index

after some logs will generatet and comming over filebeat -> logstash -> elasticsearch new index will create -> templates [] found in the elasticsearch.log

[2019-04-18T11:16:10,531][INFO ][o.e.c.m.MetaDataCreateIndexService] [devubun1604] [logstash_v00005-2019.04.18] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2019-04-18T11:16:10,681][INFO ][o.e.c.m.MetaDataMappingService] [devubun1604] [logstash_v00005-2019.04.18/7ow3RV_lS96GB65Dq0ju9g] create_mapping [_doc]
[2019-04-18T11:16:56,751][INFO ][o.e.c.m.MetaDataMappingService] [devubun1604] [logstash_v00005-2019.04.18/7ow3RV_lS96GB65Dq0ju9g] update_mapping [_doc]
localhorst-org commented 5 years ago

and in a kind of opposite, delete index again stop creating new one with stop filebeat

:~# systemctl stop filebeat

:~# curl -XDELETE 'http://localhost:9200/logstash_v00005-2019.04.18?pretty'
{
  "acknowledged" : true
}

overrite template by hand

:~# curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9200/_template/logstash_v00005?pretty' -d@elasticsearch-template_v00005.json
{
  "acknowledged" : true
}
:~# tail -f /var/log/elasticsearch/elasticsearch.log
[2019-04-18T11:28:19,301][INFO ][o.e.c.m.MetaDataIndexTemplateService] [devubun1604] adding template [logstash_v00005] for index patterns [logstash_v00005-*]

start filebeat and create index from the log incomming :~# systemctl start filebeat

:~# tail -f /var/log/elasticsearch/elasticsearch.log
[2019-04-18T11:29:45,426][INFO ][o.e.c.m.MetaDataCreateIndexService] [devubun1604] [logstash_v00005-2019.04.18] creating index, cause [auto(bulk api)], templates [logstash_v00005], shards [1]/[0], mappings [_doc]
[2019-04-18T11:29:45,601][INFO ][o.e.c.r.a.AllocationService] [devubun1604] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash_v00005-2019.04.18][0]] ...]).
[2019-04-18T11:29:45,669][INFO ][o.e.c.m.MetaDataMappingService] [devubun1604] [logstash_v00005-2019.04.18/dsGpligxQyuyhN1MRL1Taw] update_mapping [_doc]
:~# curl -s localhost:9200/_cat/indices?v
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logstash_v00005-2019.04.18      dsGpligxQyuyhN1MRL1Taw   1   0         10            0    113.4kb        113.4kb

from my side worked now like expected

krishofmans commented 5 years ago

Ran into the same (confusing) issue, turned ilm off as a workaround, thanks.

localhorst-org commented 5 years ago

today with a test installation from elk-stack 7.0.1 the problem looks solved for me. without any further configuartion changing works as expected.

thx