I'm running an application that reads from 4 medium-sized (few hundred GB) Hudi MoR tables which are compacted weekly.
When running incremental queries to look over 3~5 commits with hoodie.datasource.read.incr.fallback.fulltablescan.enable -> true, I am seeing a 5 hour gap in my Spark History Server:
When I do incremental read with hoodie.datasource.read.incr.fallback.fulltablescan.enable -> false, I see the incremental query gap go from 5 hours -> 5 mins 😮 :
Describe the problem you faced
I'm running an application that reads from 4 medium-sized (few hundred GB) Hudi MoR tables which are compacted weekly.
When running incremental queries to look over 3~5 commits with![image](https://github.com/apache/hudi/assets/24283126/8ffbb77a-79e3-4112-8d25-70069c5f2370)
hoodie.datasource.read.incr.fallback.fulltablescan.enable -> true
, I am seeing a 5 hour gap in my Spark History Server:Incremental read config:
During the gap, my driver kills all 40 of my executor instances and I see the following logs on driver: log.txt
When I use snapshot query + filter on _hoodie_commit_time, the gap reduces to ~20mins:![image](https://github.com/apache/hudi/assets/24283126/138239ba-e2a8-4d3f-837c-727e4540c20d)
Snapshot read config:
When I do incremental read with![image](https://github.com/apache/hudi/assets/24283126/3949a3c9-8b1c-485d-92d6-7b1be825ae16)
hoodie.datasource.read.incr.fallback.fulltablescan.enable -> false
, I see the incremental query gap go from 5 hours -> 5 mins 😮 :Incremental read config (no fulltable fallback):
This leads me to believe the incremental query was degenerating into a full table scan when it shouldn't... (all my commits are active and uncleaned): https://github.com/apache/hudi/blob/7a6543958368540d221ddc18e0c12b8d526b6859/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala#L142-L151
Wondering if someone can help me understand the following:
Expected behavior
Only fall back to full table scan in incremental load when necessary (file not found issue) and not all the time.
Environment Description
Hudi version : 0.13.1
Spark version : 3.4.0
Hive version : 3.1.3
Hadoop version : 3.3.3
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no