search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.15k stars 24.84k forks source link

[Logging] Support for ECS and custom event tags #46119

Closed pgomulka closed 4 years ago

pgomulka commented 5 years ago

To support Elastic Common Schema, some of the fields would have to be renamed and some possibly added to Elasticsearch JSON logs. There is a project containing a log4j2 layout that can be used to make this task easier. https://github.com/elastic/java-ecs-logging

Also since this would require new fields being added, we should consider making it easier to add more fields in the future to support "special log events" that would make monitoring of Elasticsearch easier. Example would be a special log event when cluster state changes. Draft was done here https://github.com/elastic/elasticsearch/pull/44336

elasticmachine commented 5 years ago

Pinging @elastic/es-core-infra

pgomulka commented 5 years ago

changes between current format and ECS timestamp -> @timestamp level -> log.level component -> log.logger also have a different format org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction vs previously o.e.d.r.a.a.i.RestCreateIndexAction

Additional fields from ES are not a problem, but could be migrated. These could be be part of service. set? so for instance service.node.name node.name node.id cluster.name cluster.uuid

we would map these to the custom fields. The changes from the fields above would apply Search Slow logs Index Slow logs Deprecation logs

what other fields should we add to our logs to make it more functional once parsed?

additional fields added by ECSLayout service.name -> would be cluster name? process.thread.name -> elasticsearch[node-0][masterService#updateTask][T#1]

ECSLayout defines exceptions these way. "error.code": "java.lang.IllegalArgumentException", "error.message": "persistent setting [cluster.routing.allocation.e5nable], not recognized", "error.stack_trace": array of strings (lines of stactrace

considering moving markers to json fields

cachedout commented 5 years ago

cc: @elastic/stack-monitoring for visibility

felixbarny commented 5 years ago

ECS specifies service.name and service.id. In the docs, it says this:

In the case of Elasticsearch the service.name could contain the cluster name. For Beats the service.name is by default a copy of the service.type field if no name is specified.

pgomulka commented 5 years ago

sample ECS json log line

{"@timestamp":"2019-09-20T10:54:39.539Z", "log.level": "INFO", "message":"adding index lifecycle policy [watch-history-ilm-policy]", "service.name":"ES_ECS","process.thread.name":"elasticsearch[node-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","type":"console","node.id":"suUibtbUTW6fkc5IyaGpNg","node.name":"node-0","cluster.uuid":"4GbuHHtrR4Klf_u8Hmdsng"}

with exception

{"@timestamp":"2019-09-20T11:01:09.040Z", "log.level": "WARN", "message":"path: /_cluster/settings, params: {}", "service.name":"ES_ECS","process.thread.name":"elasticsearch[node-0][http_server_worker][T#3]","log.logger":"rest.suppressed","type":"console","node.id":"_oWcocLVQCOe8ExcWMwZMg","node.name":"node-0","cluster.uuid":"Wz-9LsTlS2mgpEL2yaxhLA","error.code":"com.fasterxml.jackson.core.JsonParseException","error.message":"Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@f835b9d; line: 3, column: 36]","error.stack_trace":[
 "com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in name\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@f835b9d; line: 3, column: 36]",
 "\tat com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702)",
 "\tat com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558)",
 "\tat com.fasterxml.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:522)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:1963)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseLongName(UTF8StreamJsonParser.java:1860)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName2(UTF8StreamJsonParser.java:1840)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1797)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1732)",
 "\tat com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:776)",
 "\tat org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readGenericMap(AbstractXContentParser.java:335)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:322)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:375)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.lambda$readMap$0(AbstractXContentParser.java:322)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readGenericMap(AbstractXContentParser.java:342)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:322)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:302)",
 "\tat org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:266)",
 "\tat org.elasticsearch.rest.action.admin.cluster.RestClusterUpdateSettingsAction.prepareRequest(RestClusterUpdateSettingsAction.java:58)",
 "\tat org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:87)",
 "\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:58)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:570)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:579)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:560)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:510)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:404)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)",
 "\tat org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:374)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$3(CachingUsernamePasswordRealm.java:175)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$0(ReservedRealm.java:110)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:202)",
 "\tat org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:88)",
 "\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166)",
 "\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:365)",
 "\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:408)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:335)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:345)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:288)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:359)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:269)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:252)",
 "\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)",
 "\tat org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:390)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:306)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:317)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:244)",
 "\tat org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122)",
 "\tat org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55)",
 "\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:220)",
 "\tat org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:293)",
 "\tat org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:164)",
 "\tat org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322)",
 "\tat org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372)",
 "\tat org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301)",
 "\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69)",
 "\tat org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31)",
 "\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)",
 "\tat io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)",
 "\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)",
 "\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)",
 "\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)",
 "\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)",
 "\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)",
 "\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)",
 "\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:597)",
 "\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:551)",
 "\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)",
 "\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)",
 "\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)",
 "\tat java.base/java.lang.Thread.run(Thread.java:835)"]}
pgomulka commented 4 years ago

closed by #47105