didi / KnowStreaming

一站式云原生实时流数据平台,通过0侵入、插件化构建企业级Kafka服务,极大降低操作、存储和管理实时流数据门槛
https://knowstreaming.com
GNU Affero General Public License v3.0
6.9k stars 1.28k forks source link

How to control the size of data collected in mysql #501

Closed moluzhui closed 2 years ago

moluzhui commented 2 years ago

With the passage of time, logikm will store more and more monitoring data in mysql. Is there a mechanism to limit the data size, such as regularly deleting monitoring data?
If not, I would like to ask which of the following tables are involved in monitoring collection, and I would like to delete some monitoring data periodically such as three months to reduce the size of the data. Such as DELETE FROM cluster_metrics WHERE gmt_create BETWEEN 'xxxxxx' AND 'xxxxxxx'

Looking forward to your reply.

show tables;
+------------------------------+
| Tables_in_logi_kafka_manager |
+------------------------------+
| account                      |
| app                          |
| authority                    |
| broker                       |
| broker_metrics               |
| cluster                      |
| cluster_metrics              |
| cluster_tasks                |
| config                       |
| controller                   |
| gateway_config               |
| heartbeat                    |
| kafka_acl                    |
| kafka_bill                   |
| kafka_file                   |
| kafka_user                   |
| logical_cluster              |
| monitor_rule                 |
| operate_record               |
| reassign_task                |
| region                       |
| topic                        |
| topic_app_metrics            |
| topic_connections            |
| topic_expired                |
| topic_metrics                |
| topic_report                 |
| topic_request_time_metrics   |
| topic_statistics             |
| topic_throttled_metrics      |
| work_order                   |
+------------------------------+
ZQKC commented 2 years ago

You can config it in application.yml, and the name ends with "_metrics" are monitoring data table.

https://github.com/didi/LogiKM/blob/master/kafka-manager-web/src/main/resources/application.yml

# 任务相关的配置
task:
  op:
    sync-topic-enabled: false                       # 未落盘的Topic定期同步到DB中
    order-auto-exec:                                # 工单自动化审批线程的开关
      topic-enabled: false                          # Topic工单自动化审批开关, false:关闭自动化审批, true:开启
      app-enabled: false                            # App工单自动化审批开关, false:关闭自动化审批, true:开启
  metrics:
    collect:                                # 收集指标
      broker-metrics-enabled: true          # 收集Broker指标
    sink:                                       # 上报指标
      cluster-metrics:                          # 上报cluster指标
        sink-db-enabled: true                   # 上报到db
      broker-metrics:                           # 上报broker指标
        sink-db-enabled: true                   # 上报到db
    delete:                                           # 删除指标
      delete-limit-size: 1000                         # 单次删除的批大小
      cluster-metrics-save-days: 14                   # 集群指标保存天数
      broker-metrics-save-days: 14                    # Broker指标保存天数
      topic-metrics-save-days: 7                      # Topic指标保存天数
      topic-request-time-metrics-save-days: 7         # Topic请求耗时指标保存天数
      topic-throttled-metrics-save-days: 7            # Topic限流指标保存天数
      app-topic-metrics-save-days: 7                  # App+Topic指标保存天数