apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.43k stars 955 forks source link

[core] Add batch add partitions in MetastoreClient #4480

Closed Zouxxyy closed 1 week ago

Zouxxyy commented 2 weeks ago

Purpose

hiveClient support client.add_partitions to add partitions at once, we can use it in AddPartitionCommitCallback

this speeds up the load table a lot when have many partitions to sync.

Tests

API and Format

Documentation

wwj6591812 commented 2 weeks ago

+1