apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.43k stars 954 forks source link

[core] Add cache for snapshots and schemas #4565

Open JingsongLi opened 17 hours ago

JingsongLi commented 17 hours ago

Purpose

Considering that both snapshot and schema files are immutable, we are considering adding cache to them. Currently, we have added cache with static variables, but this carries risks, such as the following two situations:

  1. Drop the table and create a new table with same name again.
  2. Branch fastForward.

Whether these two situations are dangerous enough needs to be evaluated.

Tests

API and Format

Documentation