apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.57k stars 1.66k forks source link

[Bug] [Zeta][IMAP] can't restore job when master node down #6886

Open liunaijie opened 1 month ago

liunaijie commented 1 month ago

Search before asking

What happened

The master node shutdown because memory exceed the limit. Another node starting the mastership claim process, and become the master node. When this node initCoordinatorService, it very slow, can't restart the cluster. Here are some new master log screenshots

image

image image

Here is the old master node log screenshots

image

And i check the code, when a node became to master, it will call initCoordinatorService method.

image

in this method, it will

  1. get IMAP
  2. create JobHistoryService
  3. re-submit running jobs

In the log, i can't see the log about begin restore job (%s) from master active switch and the 2 create JobHistoryService doesn't have any operations

So i think is get IMAP fail, slow.

SeaTunnel Version

2.3.4

SeaTunnel Config

na

Running Command

na

Error Exception

na

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.