apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.29k stars 1.16k forks source link

[Feature][Extensions] Add Monitor Server #4904

Closed peacewong closed 9 months ago

peacewong commented 12 months ago

Search before asking

Problem Description

There are many Linkis clusters, with dozens of cluster nodes. In the future, it will continue to grow as the business increases. Once one of the service nodes is overloaded and has insufficient resource memory, it may cause the server to crash, which will have a certain impact on user use. The purpose of adding a monitor service is to monitor server resource usage and other indicators, and to reduce server pressure and database pressure through related scheduled cleanup tasks.

Linkis 集群众多,集群节点达到几十个。后续会随着业务增多会持续变大,一旦其中某个服务节点出现过载、资源内存不足,可能会导致服务器宕机,对用户使用会造成一定影响。增加monitor服务,其目的是为了监控服务器使用资源情况,以及对其他一些指标的监控,并且通过相关定时清理任务,来减轻服务器压力和数据库的压力。

Description

Complete monitoring of ECM resources and report indicators to IMS Complete the queuing and running count monitoring of Entrance tasks Complete monitoring of JobHistory record scanning (error codes (error codes that administrators are concerned about), task execution timeout, tenant labels, task execution result notifications) Complete the scheduled scheduling of cleanup scripts (bml historical version cleanup, task result set cleanup, task log cleanup, historical task cleanup)

完成ECM资源的监控和指标上报至IMS 完成Entrance任务的排队和运行数监控 完成JobHistory记录扫描的监控(错误码(管理员关注的错误码),任务执行超时,租户标签,任务执行结果通知) 完成清理脚本的定时调度(bml历史版本清理、任务结果集清理、任务日志清理、历史任务清理)

Use case

No response

Solutions

No response

Anything else

No response

Are you willing to submit a PR?

github-actions[bot] commented 12 months ago

:blush: Welcome to the Apache Linkis community!!

We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

Community

WeChat Assistant WeChat Public Account

Mailing Lists

Name Description Subscribe Unsubscribe Archive
dev@linkis.apache.org community activity information subscribe unsubscribe archive