GoogleCloudDataproc / flink-bigquery-connector

BigQuery integration to Apache Flink's Table API
Apache License 2.0
15 stars 11 forks source link

Committer for exactly once sink #157

Closed jayehwhyehentee closed 3 days ago

jayehwhyehentee commented 1 week ago

Invoked BQ storage flush API for commit operation.

For context, flow of execution is as follows: Checkpoint (n) -> SinkWriter.write (for all records between checkpoints n and n+1) -> SinkWriter.flush (for data accumulated in last unsent append request) -> Checkpoint (n+1) -> Commit (for data appended between checkpoints n and n+1)

While writing, SinkWriter appends data to BQ buffered stream, and uses the FlushRows API to commit the data buffered between two checkpoints.

/gcbrun

jayehwhyehentee commented 1 week ago

/gcbrun

jayehwhyehentee commented 1 week ago

@clmccart @agrawal-siddharth kindly review this PR

prashastia commented 1 week ago

/gcbrun