Pruned partition filters were not being used while doing prefix lookup for functional index records. This PR fixes that and also adds a minor optimization for getting pruned partitions and file names in a single iteration in SparkBaseIndexSupport. Added a test to validate the partition filter being used.
Impact
Optimization for functional index reads.
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.
hudi-bot
commented
2 weeks ago
Change Logs
Pruned partition filters were not being used while doing prefix lookup for functional index records. This PR fixes that and also adds a minor optimization for getting pruned partitions and file names in a single iteration in
SparkBaseIndexSupport. Added a test to validate the partition filter being used.Impact
Optimization for functional index reads.
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
Contributor's checklist