Open holdenk opened 4 months ago
We can do this by either logging the partitions that we're updating and limit comparison to those OR checking the iceberg metadata
This is important for supporting updates to large existing tables
Or we can use incremental reads if we don't care about overwrite rows -- https://iceberg.apache.org/docs/latest/spark-queries/#incremental-read
We can do this by either logging the partitions that we're updating and limit comparison to those OR checking the iceberg metadata
This is important for supporting updates to large existing tables