delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
343 stars 73 forks source link

Checkpoint file creation is skipped on restart #184

Open morgan-soulie opened 2 weeks ago

morgan-soulie commented 2 weeks ago

I am encountering an issue with the checkpoint file creation process within kafka-delta-ingest. Specifically, the problem arises when this task fails and the application exits unexpectedly. Upon restarting the application, it seems to skip the checkpoint creation step that previously failed and continues with the ingestion process as if nothing happened.