search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.52k stars 1.29k forks source link

Support checkpointing in Pinot Flink Connector #9997

Open bobby-richard-sed opened 1 year ago

bobby-richard-sed commented 1 year ago

Current the PinotSinkFunction implements the CheckpointedFunction interface, but just throws an exception during checkpointing. This make it impossible to use the pinot connector in any Flink job that requires checkpointing. I believe checkpointing could be supported in the PinotSinkFunction but storing all segment records in Flink state until the segment is committed to Pinot.

subkanthi commented 1 year ago

Its probably because of this

  @Override
  public void initializeState(FunctionInitializationContext functionInitializationContext)
      throws Exception {
    // no initialization needed
    // ...
  }

Would like to take this on.

bobby-richard-sed commented 1 year ago

Its probably because of this

  @Override
  public void initializeState(FunctionInitializationContext functionInitializationContext)
      throws Exception {
    // no initialization needed
    // ...
  }

Would like to take this on.

@subkanthi Yes, I already have a branch where I've added checkpointing support. Just wrapping up an integration test for it.