Closed soumilshah1995 closed 4 months ago
By default the Flink hudi bundle does not include any Hive related jar in dependencies, you need you package the jar manually following this doc: https://hudi.apache.org/docs/syncing_metastore#flink-setup
is there any jar available on mvn ?
what jar do I need can I find that on mvn ?
im running hive metastore locally and its accessible on 9083
is there any jar available on mvn ?
No, you need to compile it manually.
can you help me on steps if you don't mind
is there docker setup with Hive metastore and flink and hudi I really want to learn how to setup
@soumilshah1995 We don't have docker setup for that. You can build like this - https://hudi.apache.org/docs/syncing_metastore#install
Thank you.
For future reference, would it be helpful to provide links for users to download pre-built JAR files? It would greatly simplify the process, akin to how we offer JAR files for Spark. Having similar resources available for Flink and Hive would be immensely beneficial.
Thanks for the feedback, the reason that we do not include the hive into Flink Hudi bundle jar is because Flink classloader and jar conflicts is more complex, there is high possibility the hive jar can conflict with the other ones.
gotit make sense :D
Hello,
I'm currently working on creating comprehensive content for our community, focusing on integrating Apache Flink (version 1.17.1) with Python (3.8.19). While building the setup, I've encountered an issue that requires assistance from the community.
Setup:
Apache Flink version: 1.17.1 Python version: 3.8.19 Steps to Reproduce:
I've provided a Docker Compose file along with the necessary configurations to replicate the setup. Below are the essential components included in the Docker Compose file:
Issue:
The problem arises when I include the following three parameters related to Hive sync in my Flink setup:
With these parameters included, I'm unable to insert records into the system. However, when I remove these parameters, the system functions properly, allowing record insertion.
Additional Context:
This setup has been previously tested for Delta Streamer, as documented here. The goal is to achieve similar functionality with Apache Flink instead of Spark.
Code Snippet:
I've provided a full code snippet demonstrating the setup and execution process. This includes defining the Hudi table, executing SQL queries, and attempting to insert records into the Hudi table.
Request for Assistance:
I'm seeking assistance from the community to understand how to properly configure Hive sync parameters in my Flink setup. Any guidance or insights would be greatly appreciated.