Open lightzhao opened 1 month ago
If we modify the logic like this, is it possible that the Kafka cluster is abnormal, but SeaTunnel cannot exit normally?
Yes, the expectation is that the consumer thread will not exit after the Kafka cluster fails, and continue to execute after recovery, otherwise all tasks will need to be restarted
If we modify the logic like this, is it possible that the Kafka cluster is abnormal, but SeaTunnel cannot exit normally?
Yes, the expectation is that the consumer thread will not exit after the Kafka cluster fails, and continue to execute after recovery, otherwise all tasks will need to be restarted
In this case, users cannot use the SeaTunnel job status to determine whether the data reading is running normally. This is unacceptable
In this case, users cannot use the SeaTunnel job status to determine whether the data reading is running normally. This is unacceptable
OK, SeaTunnel expects that after the kafka cluster fails, it should be restarted to recover, right?
In this case, users cannot use the SeaTunnel job status to determine whether the data reading is running normally. This is unacceptable
OK, SeaTunnel expects that after the kafka cluster fails, it should be restarted to recover, right?
It should be that after the Kafka cluster fails, it means that the synchronization job cannot continue to run normally, so the job should fail.
In this case, users cannot use the SeaTunnel job status to determine whether the data reading is running normally. This is unacceptable
OK, SeaTunnel expects that after the kafka cluster fails, it should be restarted to recover, right?
It should be that after the Kafka cluster fails, it means that the synchronization job cannot continue to run normally, so the job should fail.
OK, but the current mechanism is that the consumer thread just exits, the job does not fail and continues to run. I think this is unreasonable.
In this case, users cannot use the SeaTunnel job status to determine whether the data reading is running normally. This is unacceptable
OK, SeaTunnel expects that after the kafka cluster fails, it should be restarted to recover, right?
It should be that after the Kafka cluster fails, it means that the synchronization job cannot continue to run normally, so the job should fail.
OK, but the current mechanism is that the consumer thread just exits, the job does not fail and continues to run. I think this is unreasonable.
+1 I focus this problem, too
OK, but the current mechanism is that the consumer thread just exits, the job does not fail and continues to run. I think this is unreasonable.
Yes, it's a bug. Could you help to fix it?
OK, but the current mechanism is that the consumer thread just exits, the job does not fail and continues to run. I think this is unreasonable.
Yes, it's a bug. Could you help to fix it?
Of course, but what strategies should we adopt after a Kafka cluster failure?
Of course, but what strategies should we adopt after a Kafka cluster failure?
The job should be failed.
Of course, but what strategies should we adopt after a Kafka cluster failure?
The job should be failed.
ok, I will fix.
Of course, but what strategies should we adopt after a Kafka cluster failure?
The job should be failed.
When the Kafka cluster fails, KafkaConsumer.poll will not throw any related exceptions, but will only log them. Therefore, the task side cannot get the exception to make it fail. It is recommended not to exit the thread and keep it running.
When the Kafka cluster fails, KafkaConsumer.poll will not throw any related exceptions, but will only log them. Therefore, the task side cannot get the exception to make it fail. It is recommended not to exit the thread and keep it running.
Ok for me. cc @hailin0
Purpose of this pull request
When the Kafka cluster or coordinator fails, it will cause a connection exception. After catching the exception, it should not be thrown, which will cause the Kafka consumer thread to exit. When the Kafka cluster is restored, the task can no longer be consumed.
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
release-note
.