search

trentm / go-ecslog

`ecslog` CLI to pretty-print and filter log files in ecs-logging format
Apache License 2.0
7 stars 2 forks source link

elasticsearch logs often don't have ecs.version: cope or punt? #11

Open trentm opened 3 years ago

trentm commented 3 years ago 
{"@timestamp":"2021-04-08T16:22:52.257Z", "log.level": "INFO", "message":"adding index lifecycle policy [apm-rollover-30-days]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[01f5411cc2d6][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","event.dataset":"elasticsearch.server","elasticsearch.cluster.uuid":"wMGQhNeoRKO0HO0TdVWqtw","elasticsearch.node.id":"-A230e5wSk6Pubf3TrFhzw","elasticsearch.node.name":"01f5411cc2d6","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2021-04-08T16:22:52.296Z", "log.level": "INFO", "message":"adding template [apm-8.0.0-metric] for index patterns [apm-8.0.0-metric*]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[01f5411cc2d6][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","event.dataset":"elasticsearch.server","elasticsearch.cluster.uuid":"wMGQhNeoRKO0HO0TdVWqtw","elasticsearch.node.id":"-A230e5wSk6Pubf3TrFhzw","elasticsearch.node.name":"01f5411cc2d6","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2021-04-08T16:22:52.326Z", "log.level":"DEPRECATION",  "data_stream.dataset":"elasticsearch.deprecation", "data_stream.namespace":"default", "data_stream.type":"logs", "ecs.version":"1.7", "elasticsearch.event.category":"templates", "event.code":"index_template_multiple_match", "message":"index [apm-8.0.0-metric-000001] matches multiple legacy templates [apm-8.0.0, apm-8.0.0-metric], composable templates will only match a single template" , "service.name":"ES_ECS","process.thread.name":"elasticsearch[01f5411cc2d6][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.cluster.metadata.MetadataCreateIndexService","event.dataset":"elasticsearch.deprecation","elasticsearch.cluster.uuid":"wMGQhNeoRKO0HO0TdVWqtw","elasticsearch.node.id":"-A230e5wSk6Pubf3TrFhzw","elasticsearch.node.name":"01f5411cc2d6","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2021-04-08T16:22:52.344Z", "log.level": "INFO", "message":"[apm-8.0.0-metric-000001] creating index, cause [api], templates [apm-8.0.0-metric, apm-8.0.0], shards [1]/[0]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[01f5411cc2d6][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataCreateIndexService","event.dataset":"elasticsearch.server","elasticsearch.cluster.uuid":"wMGQhNeoRKO0HO0TdVWqtw","elasticsearch.node.id":"-A230e5wSk6Pubf3TrFhzw","elasticsearch.node.name":"01f5411cc2d6","elasticsearch.cluster.name":"docker-cluster"}
{"@timestamp":"2021-04-08T16:22:52.399Z", "log.level": "INFO", "message":"moving index [apm-8.0.0-metric-000001] from [null] to [{\"phase\":\"new\",\"action\":\"complete\",\"name\":\"complete\"}] in policy [apm-rollover-30-days]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[01f5411cc2d6][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.IndexLifecycleTransition","event.dataset":"elasticsearch.server","elasticsearch.cluster.uuid":"wMGQhNeoRKO0HO0TdVWqtw","elasticsearch.node.id":"-A230e5wSk6Pubf3TrFhzw","elasticsearch.node.name":"01f5411cc2d6","elasticsearch.cluster.name":"docker-cluster"}

^^ some sample logs from an ES 8.0.0 container of mine. Only a small fraction of log lines have a "ecs.version" field. Should ecslog support an option to accept those? or punt and open a ticket on ES?

trentm commented 3 years ago

Note to self, total hack for play for now:

docker logs localtesting_8.0.0_elasticsearch -f | json -ga -o jsony-0 -e 'this["ecs.version"]="1"' | ecslog