apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.47k stars 2.43k forks source link

[SUPPORT] sparkSql create table sync metastore Failed #9575

Closed viverlxl closed 1 year ago

viverlxl commented 1 year ago

Tips before filing an issue

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

  1. SparkSql Client: xxx/spark-3.1.2-bin-hadoop3.2/bin/spark-sql --jars XX/hudi-spark-bundle_2.11-0.10.0.jar --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
  2. create table hudi_mor_tbl ( id int, name string, price double, ts bigint ) using hudi tblproperties ( type = 'mor', primaryKey = 'id', preCombineField = 'ts' );
  3. WARN [main] config.DFSPropertiesConfiguration (DFSPropertiesConfiguration.java:getConfPathFromEnv(216)) - Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf 2023-08-30 16:12:37,279 WARN [main] config.DFSPropertiesConfiguration (DFSPropertiesConfiguration.java:addPropsFromFile(150)) - Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file 2023-08-30 16:12:40,660 WARN [main] command.CreateHoodieTableCommand (Logging.scala:logWarning(69)) - Failed to create catalog table in metastore: org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat

Expected behavior hive can find table which create

A clear and concise description of what you expected to happen.

Environment Description

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

related: https://github.com/apache/hudi/issues/6128

had add jar to hive but no work

viverlxl commented 1 year ago

https://github.com/apache/hudi/issues/1646

viverlxl commented 1 year ago

solved