Open soumilshah1995 opened 4 months ago
@soumilshah1995 You are right. when the number of cleaner commits is less than 20, it actually compares the archival commits with 20 only. When it's more than 20 then it is behaving fine. Though the code just directly compares the same -
Need to take a deeper look why can this happen
Roger that sir
Issue Description: I'm encountering an issue with Hudi configurations related to commit retention and cleaning. Despite explicitly setting hoodie.cleaner.commits.retained to 5, I'm receiving a warning suggesting it should be set to 20. It seems like the system is not acknowledging my provided value and is using some default instead.
Warning Message:
Steps to Reproduce: Use the above Hudi configuration. Run the ingestion process using the provided code sample. Expected Behavior: The system should respect the explicitly set hoodie.cleaner.commits.retained value of 5 without suggesting an increase to 20.
Actual Behavior: The system issues a warning to increase hoodie.keep.min.commits to be greater than hoodie.cleaner.commits.retained set to 20, despite hoodie.cleaner.commits.retained being explicitly set to 5.
Environment: Hudi Version: 0.14.0 Spark Version: 3.4 OS: macOS Code Sample:
Additional Context: This behavior seems to indicate an internal adjustment or default setting that overrides the user-defined configuration, potentially causing confusion and misconfiguration. Any insights or fixes to ensure that the provided configurations are respected would be greatly appreciated.