dianping / cat

CAT 作为服务端项目基础组件,提供了 Java, C/C++, Node.js, Python, Go 等多语言客户端,已经在美团点评的基础架构中间件框架(MVC框架,RPC框架,数据库框架,缓存框架等,消息队列,配置系统等)深度集成,为美团点评各业务线提供系统丰富的性能指标、健康状况、实时告警等。
Apache License 2.0
18.69k stars 5.43k forks source link

告警延迟分析 #2243

Closed romanticmj closed 1 year ago

romanticmj commented 2 years ago

背景 微服务>50,告警>2w时,经常出现告警延迟,最多时候达到1h以上

原因分析 SendExcutor每5min拉取一次告警,通过arthas分析,主要是在 m_decoratorManager.generateTitleAndContent 耗时长,发现是其中有一条SQL执行很慢 SELECT al.id,al.domain,al.alert_time,al.category,al.type,al.content,al.metric,al.creation_date FROM alert al WHERE al.alert_time >= ? AND al.alert_time <= ? AND al.category = ? AND al.domain = ? ORDER BY al.alert_time asc

解决策略 增加索引index idx_alert_time_category_domain(alert_time,category,domain); alter tablealertadd index idx_alert_time_category_domain(alert_time,category,domain);

建议 在初始化的CatApplication.sql中,补充该索引

qmwu2000 commented 2 years ago

麻烦顺便发个pull request,谢谢!

On Aug 15, 2022, at 14:48, romanticmj @.***> wrote:

背景 微服务>50,告警>2w时,经常出现告警延迟,最多时候达到1h以上

原因分析 SendExcutor每5min拉取一次告警,通过arthas分析,主要是在 m_decoratorManager.generateTitleAndContent 耗时长,发现是其中有一条SQL执行很慢 SELECT al.id,al.domain,al.alert_time,al.category,al.type,al.content,al.metric,al.creation_date FROM alert al WHERE al.alert_time >= ? AND al.alert_time <= ? AND al.category = ? AND al.domain = ? ORDER BY al.alert_time asc

解决策略 增加索引index idx_alert_time_category_domain(alert_time,category,domain); alter table alert add index idx_alert_time_category_domain(alert_time,category,domain);

建议 在初始化的CatApplication.sql中,补充该索引

— Reply to this email directly, view it on GitHub https://github.com/dianping/cat/issues/2243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASQE73KM2TVQSE5SMT6QRDVZHR2BANCNFSM56RIPGQQ. You are receiving this because you are subscribed to this thread.

romanticmj commented 2 years ago

PR:https://github.com/dianping/cat/pull/2245

qmwu2000 commented 2 years ago

Merged into master

On Aug 17, 2022, at 10:31, romanticmj @. @.>> wrote:

PR:#2245 https://github.com/dianping/cat/pull/2245 — Reply to this email directly, view it on GitHub https://github.com/dianping/cat/issues/2243#issuecomment-1217387424, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASQE77BRS2M5PA6YTRIE2TVZRFJ3ANCNFSM56RIPGQQ. You are receiving this because you commented.