[HUDI-4468] Simplify TimeTravel logic for Spark

KnightChess commented 4 days ago

Change Logs

After spark3.3 support TimeTravel SQL and Logical Plan, remove the implementation in hudi, remain index DML SQL

code detail: before spark3.3, spark not support timetravel sql, so hudi supports the full DQL query syntax for timetravel. now, we remove it, therefore, a large portion of the DQL syntax parsing will not be used; we only need to support Hudi's custom implementation of the DDL syntax related to indexing.

g4 file

delete queryStatement dmlStatement createTable in HoodieSqlBase.g4, it's for timetravel
delete SqlBase.g4 file, copy which index need.

astBuilder

delete a large amount of redundant implementations and retain only those related to indexing

Impact

in spark, TIMESTAMP dataType is TimestampType, VERSION is String. And the hoodie instance time format is not legal for spark to parse. And from the perspective of usage, the concept of HUDI's timestamp concept is more suitable for using VERSION, so in this pr:

VERSION AS OF
- support digit string, contain hudi base timeline format - [yyyyMMddhhmmssSSS], will return less or equal snapshots
TIMSTAMP AS OF
- spark provides a rich date expression, will return less or equal snapshots

Risk level (write none, low medium or high below)

low

Documentation Update

spark Time Travel Query tutorial, update

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

vinothchandar commented 3 days ago

@jonvex can you please review this

jonvex commented 3 days ago

@KnightChess can you please explain what changes you made to the files with 1000+ added or removed lines of code. Thanks!

danny0405 commented 4 hours ago

VERSION AS OF support digit string, contain hudi base timeline format - [yyyyMMddhhmmssSSS], will return less or equal snapshots TIMSTAMP AS OF spark provides a rich date expression, will return less or equal snapshots

@KnightChess do we add new syntax or change existing syntax in the patch?

hudi-bot commented 3 hours ago

CI report:

00a6e05dd4cb28236fe067ead487d5427ad51734 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build

apache / hudi