search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.35k stars 928 forks source link

[flink] Support cache lookup operator in flink #3522

Closed FangYongs closed 1 month ago

FangYongs commented 3 months ago

Purpose

Linked issue: close #3521

Support flink cache lookup operator

Tests

Added test case LookupJoinITCase#testLookupTableWithCache

API and Format

no

Documentation

no

JingsongLi commented 3 months ago

Hi @FangYongs , can you explain why we should use flink cache? Any performance benifits?

FangYongs commented 2 months ago

@JingsongLi We want to reuse metrics in flink job such cache hits and misses, and we will add more metrics such as qps/latency in flink lookup operator. WDYT?

JingsongLi commented 2 months ago

@JingsongLi We want to reuse metrics in flink job such cache hits and misses, and we will add more metrics such as qps/latency in flink lookup operator. WDYT?

Add more metrics to paimon lookup function looks good to me~

JingsongLi commented 1 month ago

Close this now, we can just add more metrics.