ELK: 使用ELK进行日志采集

[lihongjie0209]

整体的架构

1. 使用filebeat把日志文件写入es

[lihongjie0209]

安装filebeat

 yum install -y filebeat

配置filebeat

Modify filebeat.yml to set the connection information:

output.elasticsearch:
  hosts: ["<es_url>"]
  username: "elastic"
  password: "<password>"
setup.kibana:
  host: "<kibana_url>"

导入kibana 图表

filebeat setup

配置采集模块

filebeat modules enable system

具体的配置可以在 /etc/filebeat/modules.d/ 中编辑

启动filebeat

systemctl start filebeat

效果图

[lihongjie0209]

filebeat + logback

首先spring boot 项目配置logback

添加maven依赖用于发送日志到filebeat

    <dependency>
      <groupId>net.logstash.logback</groupId>
      <artifactId>logstash-logback-encoder</artifactId>
      <version>6.4</version>
    </dependency>
    <!-- Your project must also directly depend on either logback-classic or logback-access.  For example: -->
    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
    </dependency>

配置appender

    <appender name="tcp" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>192.168.0.115:7894</destination>

        <!-- encoder is required -->
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>
    <root level="info">
        <appender-ref ref="console" />
        <appender-ref ref="file" />
        <appender-ref ref="tcp" />
    </root>

配置filebeat监听tcp端口

- type: tcp
  host: "0.0.0.0:7894"
  index: petclinic-%{+yyyy.MM.dd}
  enabled: true
  processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: ""
        overwrite_keys: true
        add_error_key: true

processors的主要作用是把日志对象的json字符串展开, 这样方便检索.

效果展示: