search

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
919 stars 147 forks source link

How to infer partition fields from hudi source table and use them datasetConfig.yaml #411

Open rahul-ghiware opened 7 months ago

rahul-ghiware commented 7 months ago

Hi Team,

We're looking to infer partition fields (if any) associated with a given Hudi source table and incorporate them into the datasetConfig.yaml file (as a partitionSpec) before running the XTable utility jar for data format conversion.

Could someone please assist me with this?

the-other-tim-brown commented 7 months ago

Do you have access to the writer configs for these tables? You can infer based on the options used. https://hudi.apache.org/docs/key_generation

TimestampBasedKeyGenerator and CustomKeyGenerator are the ones that would require the most effort since you need to also extract the output format for the partition when it is time based.