apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.45k stars 2.42k forks source link

[HUDI-8550] Make Hudi 1.x write timeline to a dedicated timeline folder under .hoodie #12288

Open bvaradar opened 3 days ago

bvaradar commented 3 days ago

Change Logs

Hudi 1.x to use dedicated timeline folder under .hoodie

Impact

Timeline folder to move to a dedicated folder.

Risk level (write none, low medium or high below)

none

Documentation Update

none

Contributor's checklist

bvaradar commented 1 day ago

@danny0405 : Apart from the two comments, Are you ok with other changes ?

danny0405 commented 1 day ago

@danny0405 : Apart from the two comments, Are you ok with other changes ?

Yes, overall looks good to me, I reviewed EightToSevenDowngradeHandler in detail and it should be good, the other changes are minor besides that meta client refresh change.

vinothchandar commented 16 hours ago

@bvaradar Have not looked very closely. but can we also make sure the lsm timeline moves along with new folder.. Call it history instead of archived?

so

.hoodie 
   |______timeline <-- active lives here
                       |________ history <-- lsm lives here.

over 1.1 or 1.2 I want to fully converge it to a single timeline.

hudi-bot commented 2 hours ago

CI report:

Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build