search

AliyunContainerService / log-pilot

Collect logs for docker containers
https://yq.aliyun.com/articles/674327
Apache License 2.0
1.43k stars 402 forks source link

log-pilot.filebeat 收集java日志，合并多行，log-pilot能够解析，但无法存es #169

Open weihc02 opened 5 years ago

weihc02 commented 5 years ago

查看log-polit 日志，可查看能正常解析格式，如下格式。 {"@timestamp":"2019-01-04T17:24:46.920Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.1.1"},"prospector":{"type":"log"},"beat":{"hostname":"log-pilot-qmg2z","version":"6.1.1","name":"log-pilot-qmg2z"},"stream":"stdout","message":"2019-01-05 01:24:46.919 WARN 1 --- [nfoReplicator-0] o.s.b.actuate.mail.MailHealthIndicator : Mail health check failed\n\ncom.sun.mail.util.MailConnectException: Couldn't connect to host, port: smtp.servingcloud.com, 25; timeout -1\n\tat com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2209) ~[javax.mail-1.6.2.jar!/:1.6.2]\n\tat com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:740) ~[javax.mail-1.6.2.jar!/:1.6.2]\n\tat javax.mail.Service.connect(Service.java:366) ~[javax.mail-1.6.2.jar!/:1.6.2]\n\tat org.springframework.mail.javamail.JavaMailSenderImpl.connectTransport(JavaMailSenderImpl.java:515) ~[spring-context-support-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]\n\tat org.springframework.mail.javamail.JavaMailSenderImpl.testConnection(JavaMailSenderImpl.java:396) ~[spring-context-support-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]\n\tat org.springframework.boot.actuate.mail.MailHealthIndicator.doHealthCheck(MailHealthIndicator.java:43) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]\n\tat org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:84) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]\n\tat org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) [spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]\n\tat org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:103) [spring-cloud-netflix-eureka-client-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]\n\tat org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:99) [spring-cloud-netflix-eureka-client-2.0.2.RELEASE.jar!/:2.0.2.RELEASE]\n\tat com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1382) [eureka-client-1.9.3.jar!/:1.9.3]\n\tat com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117) [eureka-client-1.9.3.jar!/:1.9.3]\n\tat com.netflix.discovery.InstanceInfoReplicator$1.run(InstanceInfoReplicator.java:101) [eureka-client-1.9.3.jar!/:1.9.3]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181]\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_181]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]\nCaused by: java.net.ConnectException: Operation timed out (Connection timed out)\n\tat java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_181]\n\tat java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_181]\n\tat java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_181]\n\tat java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_181]\n\tat java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_181]\n\tat java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_181]\n\tat java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_181]\n\tat com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:359) ~[javax.mail-1.6.2.jar!/:1.6.2]\n\tat com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:238) ~[javax.mail-1.6.2.jar!/:1.6.2]\n\tat com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2175) ~[javax.mail-1.6.2.jar!/:1.6.2]\n\t... 19 common frames omitted\n","offset":70856,"topic":"xcloudmessage","k8s_container_name":"container-spacexcloud-message-app","k8s_pod":"deployment-spacexcloud-message-5c95b54d75-bk2rw","source":"/host/var/lib/docker/containers/358afeea419fb01dc8dcf15a1284e22de121e35ef300eb13e0687e24f056d80d/358afeea419fb01dc8dcf15a1284e22de121e35ef300eb13e0687e24f056d80d-json.log","docker_container":"k8s_container-spacexcloud-message-app_deployment-spacexcloud-message-5c95b54d75-bk2rw_servingcloud-dev-apps_ae134cd5-1044-11e9-8e2f-00163e103a44_0","k8s_pod_namespace":"servingcloud-dev-apps","index":"xcloudmessage"}

log-polit -filebeat版本0.9， filebeat.tpl。现在存在问题如果把多行合并方式去掉，可以在elk kibana界面看到相应的日志，不过日志被截断，如果加下面多行合并，即无法收集到相应日志，查看elk日志，也无相关报错日志 multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after multiline.max_lines: 200

chenquanzhao commented 5 years ago

@weihc02 可看下/var/log/filebat日志情况，在这种情况下

Darren-wh commented 5 years ago

这个问题解决了吗 如何合并多行的？

Linuxbody commented 5 years ago

这个问题解决了吗 ？修改多行 能解析，但是没写到 es 里面

Darren-wh commented 5 years ago

没有解决吧。建议采用sidecar的方式来收集容器日志吧。

liqilong2017 commented 4 years ago

这个问题解决了吗 ？修改多行 能解析，但是没写到 es 里面

最终解决了吗？？？？