apache / hudi-rs

A native Rust library for Apache Hudi, with bindings into Python
https://hudi.apache.org/
Apache License 2.0
149 stars 30 forks source link

Creating table instances should also load `hudi-defaults.conf` #111

Closed xushiyan closed 2 months ago

xushiyan commented 3 months ago

During crates/core/src/table/mod.rs:96 load_configs<I, K, V>, hudi global configs (a hudi-defaults.conf file loaded in dir specified by HUDI_CONF_DIR env var) should be loaded as well.

gohalo commented 3 months ago

i could work on that if no one plan for.

As externalized-config-file described , should scan all files under HUDI_CONF_DIR directory, and maybe filter with *.conf.

And i havn't found the format of this file, maybe properties, ini etc. Just got some ideas from Hudi Issues #5291, like the following.

# Default system properties included when running Hudi jobs.
# This is useful for setting default environmental settings.

# Example:
hoodie.datasource.write.table.type                      COPY_ON_WRITE
hoodie.datasource.write.hive_style_partitioning         false
xushiyan commented 3 months ago

@gohalo great! as per the linked doc, only hudi-defaults.conf should be loaded under HUDI_CONF_DIR. The format should be like a conf file as you showed or delimited by = like

hoodie.datasource.write.recordkey.field=uuid