[SUPPORT] Tutorial on registering Hudi files into Hive metastore

[torvalds-dev-testbot[bot]]

Tips before filing an issue

• Have you gone through our FAQs?
• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced A clear and concise description of the problem.

To Reproduce Steps to reproduce the behavior: 1. 2. 3. 4.

Expected behavior A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :
• Spark version :
• Hive version :
• Hadoop version :
• Storage (HDFS/S3/GCS..) :
• Running on Docker? (yes/no) :

Additional context Add any other context about the problem here.

Stacktrace Add the stacktrace of the error.

[torvalds-dev-testbot[bot]]

Relevant Sources:

https://hudi.apache.org/docs/syncing_metastore

https://medium.com/@devjain1299/apache-hudi-on-aws-glue-a-step-by-step-guide-503c34a9aa95

https://apache-hudi.slack.com/archives/C4D716NPQ/p1714088000801379

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

Follow up questions:

1. Could you describe the version of Hudi and other tech stack in your environment? If you are facing an error, please provide a full stack trace and describe what operations led to the errors. Some other artifacts that are useful to triage would be Spark UI screenshots.
2. Can you please help us understand your environment a little better? It would help to get the versions and Hudi write configs. Also, if you have not already specified, could you describe the nature of your dataset? For example, are there inserts? If so, what is the ratio of the inserts and updates?
3. Can you please share the Hudi write configs and versions of Hudi and the compute engine you are using? In addition, please provide the query you used to detect duplicates.

Sources:

• https://hudi.apache.org/docs/syncing_metastore
• https://medium.com/@devjain1299/apache-hudi-on-aws-glue-a-step-by-step-guide-503c34a9aa95
• https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-hudi.html
• https://hudi.apache.org/docs/docker_demo

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I generate the follow up questions; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

In order to register Hudi files into the Hive Metastore, you can follow these steps:

• Use the Hudi Hive Sync Tool by running ./run_sync_tool.sh with the necessary parameters like JDBC URL, database, table, partitioned-by, base path. Ensure you have built the hudi-hive module. Refer to the documentation mentioned at https://hudi.apache.org/docs/syncing_metastore for detailed guidance on using the Hive Sync Tool.

• If you want to sync Hudi tables directly to the AWS Glue Data Catalog, you can utilize the org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool available and configure it by setting --sync-tool-classes org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool. More information can be found at https://hudi.apache.org/docs/syncing_aws_glue_data_catalog.

• Ensure that the necessary configurations like database name, table name, and base path are correctly set while syncing the Hudi table to Hive. These configurations are crucial for successful synchronization, as outlined in detail at https://hudi.apache.org/docs/syncing_metastore.

By following these steps and referring to the provided documentation snippets, you can effectively sync Hudi files into the Hive Metastore and AWS Glue Data Catalog.

[torvalds-dev-testbot[bot]]

Abhishek posted:

https://devs-numberlabs.slack.com/archives/C06PDCQPA3W/p1722424382585679?thread_ts=1722424506915009&cid=C06PDCQPA3W

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/234