apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.45k stars 2.42k forks source link

[HUDI-8530] Error out secondary index update for write operations such as insert overwrite where it is not supported #12280

Closed lokeshj1703 closed 19 hours ago

lokeshj1703 commented 3 days ago

Change Logs

For overwrite related write operations like insert overwrite, insert overwrite table and delete partition operations we need to throw an error indicating that secondary index update is not supported with these operations. We would be adding the support of such operations in a separate jira. Currently the user would need to drop and recreate the index after the operation.

Impact

INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE and DELETE_PARTITION would throw error if secondary index is enabled

Risk level (write none, low medium or high below)

low

Documentation Update

INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE and DELETE_PARTITION would throw error if secondary index is enabled. Index needs to be dropped and recreated after the operation

Contributor's checklist

nsivabalan commented 1 day ago

there are some test failures. can you check it out. @lokeshj1703

codope commented 22 hours ago

I've fixed the test failures. Should be good to land after they pass.

hudi-bot commented 20 hours ago

CI report:

Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build