search

apache / incubator-streampark

Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
https://streampark.apache.org/
Apache License 2.0
3.91k stars 1.01k forks source link

[Feature] flink cluster failure alarm&failover #2423

Closed xujiangfeng001 closed 1 year ago

xujiangfeng001 commented 1 year ago

Search before asking

Description

a. If cluster shutdown or lost is detected, an alarm will be sent. b. If the job is running on the cluster, the job will alarm in batches. At this time, it is necessary to prevent the job from alarming.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

xujiangfeng001 commented 1 year ago

Hello everyone, after discussion, our solution for this issue is as follows: Requirement:

Detailed logic:

  1. Flink Cluster implements mentality detection and status updates, as detailed in: https://github.com/apache/incubator-streampark/pull/2675
  2. When an exception occurs in a job, it is necessary to determine whether the job deployment mode is remote, yarn session or k8s session：If not, send the job alarm directly. If so, obtain the flink cluster status through the Flink Cluster ID of the job:If the flink cluster status is STOP or LOST, block the job alarm and wait for the flink cluster alarm.If the status of flink cluster is RUNNING, actively trigger a flink cluster status update request to update the relevant status of flink cluster. If flink cluster is updated to STOP or LOST status in the latest update, the job alarm will be blocked; If the flink cluster status is still RUNNING, send an alarm notification for the job.
  3. Flink cluster alarm template uses job alarm template and adds information: number of affected jobs.
  4. Abstract the alarm template code to avoid code redundancy issues.