search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.35k stars 928 forks source link

[common] Minor refactor PredicateBuilder to avoid potential stack overflow #3517

Closed yuzelin closed 2 months ago

yuzelin commented 3 months ago

Purpose

The case is the upstream data changes too many partitions, make the Predicate too deep, then error is:

Caused by: java.lang.RuntimeException: java.lang.StackOverflowError
    at org.apache.paimon.manifest.ManifestFileMeta.merge(ManifestFileMeta.java:178)
    at org.apache.paimon.operation.FileStoreCommitImpl.tryCommitOnce(FileStoreCommitImpl.java:808)
    ... 27 more
Caused by: java.lang.StackOverflowError
    at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
    at org.apache.paimon.predicate.Or.test(Or.java:55)
    at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
    ...

Tests

API and Format

Documentation

JingsongLi commented 2 months ago

Finished in https://github.com/apache/paimon/pull/3864