After debugging, I found that upgrade path is invoked but the metaClient is not reloaded after that. Simply reloading in tryUpgrade is not enough because we call that in a lambda. So, this PR fixes the issue as follows:
Updated doInitTable() to return the potentially reloaded metaClient, allowing initTable() to access the updated metaClient reference after an upgrade.
Enhanced HoodieCommitMetadata.fromBytes() method with a fallback mechanism for JSON deserialization given that commit metadata serialization changed from json to avro in 1.0.
Tested manually by running the same steps as mentioned in HUDI-8395.
Impact
Refactor metaClient handling during upgrade/downgrade, and implement fallback in JSON deserialization.
Risk level (write none, low medium or high below)
medium
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.
hudi-bot
commented
1 week ago
Change Logs
After debugging, I found that upgrade path is invoked but the
metaClientis not reloaded after that. Simply reloading intryUpgradeis not enough because we call that in a lambda. So, this PR fixes the issue as follows:doInitTable()to return the potentially reloadedmetaClient, allowinginitTable()to access the updatedmetaClientreference after an upgrade.HoodieCommitMetadata.fromBytes()method with a fallback mechanism for JSON deserialization given that commit metadata serialization changed from json to avro in 1.0.Tested manually by running the same steps as mentioned in HUDI-8395.
Impact
Refactor metaClient handling during upgrade/downgrade, and implement fallback in JSON deserialization.
Risk level (write none, low medium or high below)
medium
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
Contributor's checklist