apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.35k stars 2.42k forks source link

[SUPPORT] Hudi Time Travel from Athena #7594

Open maddy2u opened 1 year ago

maddy2u commented 1 year ago

Hi,

Simple question - Does AWS Athena support Hudi Time Travel?

We see good support for Iceberg tables here I see some mention in this ticket about it but can you share more details about it?

IvyIvy1109 commented 1 year ago

Hi,

Simple question - Does AWS Athena support Hudi Time Travel?

We see good support for Iceberg tables here I see some mention in this ticket about it but can you share more details about it?

+1

tooptoop4 commented 1 year ago

https://github.com/trinodb/trino/pull/15084 hasn't been merged

maddy2u commented 1 year ago

@tooptoop4 : Any expectation on when this will be merged to Trinodb? I think it takes a while before it is made available within Athena post this.

Not sure if this is the right place to ask the question but here it goes, Is there a way today to enable Cross AWS Account Hudi datasets to have time travel capabilities?

tooptoop4 commented 1 year ago

not sure when. for 2nd qn, at least in trino itself I think u can test that PR on a local patch build, I assume it will work as long as u define the right IAM permissions

maddy2u commented 1 year ago

Thanks for the information @tooptoop4 ! I will keep an eye on the PR for updates.

yihua commented 1 year ago

@umehrot2 @rahil-c do you folks have any information on this question, around when Athena can pick up the Trino side change when the PR is merged?

soumilshah1995 commented 1 year ago

Athena now supports sparks notebook i think you can easily use that to solve your issue also you can query timestamp column and achieve that as well

Video 👍

https://www.youtube.com/watch?v=Tkltri-9s7k&t=31s

Amazon Athena now supports Apache Spark Amazon Athena now supports Apache Spark

maddy2u commented 1 year ago

Need to test it. I believe Lake Formation will not support cross account time travel yet as it's not yet supported through lake formation permissions but since Athena seems to work, I'm not sure whether this approach will work.

xushiyan commented 1 year ago

cc @jonvex