[HUDI-8490] Implicit lock provider use different lock key scheme

[Davis-Zhang-Onehouse]

Change Logs

code + test

Impact

For implicit lock provider org.apache.hudi.aws.transaction.lock.DynamoDBBasedImplicitPartitionKeyLockProvider and org.apache.hudi.client.transaction.lock.ZookeeperBasedImplicitBasePathLockProvider, inside the lock key (for zookeeper) / partition key (for ddb) it contains substring <tablename>-<hash of table basepath>.

This is problematic in case of alter table rename, some writers are using the old lock key while others are using the new one, which leads to concurrency bug as the lock fails to synchronize operations from concurrent operators.

Alter rename should not have coupling with locks, so to remove the dependency, we removed the table name from the lock key for both implicit ddb and zookeeper lock provider.

Risk level (write none, low medium or high below)

Low

Documentation Update

We should update the merge into doc about this new restriction.

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[hudi-bot]

CI report:

• fbff054719c7101a2bac3f4ae54603a889e68bcb Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build

[nsivabalan]

when was this lock provider added? if it was part fo 0.15.0, we need to add a doc update (website update) on the runbook. can you file a tracking ticket if applicable.