[SUPPORT] Support for multiple file formats in the same SparkSession

[torvalds-dev-testbot[bot]]

Tips before filing an issue

• Have you gone through our FAQs?
• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced A clear and concise description of the problem: Is it possible to support both Hudi and Delta Lake file formats in the same SparkSession for Spark 3.2 and above?

To Reproduce Steps to reproduce the behavior: 1. 2. 3. 4.

Expected behavior A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :
• Spark version : 3.2 and above
• Hive version :
• Hadoop version :
• Storage (HDFS/S3/GCS..) :
• Running on Docker? (yes/no) :

Additional context Add any other context about the problem here.

Stacktrace Add the stacktrace of the error.

[torvalds-dev-testbot[bot]]

Relevant Sources:

https://hudi.apache.org/docs/configurations

https://hudi.apache.org/docs/quick-start-guide

https://blog.datumagic.com/p/apache-hudi-from-zero-to-one-910?r=2fl10k&utm_campaign=post&utm_medium=web

https://apache-hudi.slack.com/archives/C4D716NPQ/p1713511341371689

[torvalds-dev-testbot[bot]]

Not satisified with the answer

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

Based on the provided documentation, to support both Hudi and Delta Lake in the same SparkSession, you need to use the spark_catalog configuration parameter accordingly:

• Use 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' for Hudi
• Use "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" for Delta Lake.

Follow-up questions:

• Are you using Spark 3.2 or higher versions?
• Do you encounter any specific errors or issues while attempting to support both file formats simultaneously?
• What is the primary use case for needing both Hudi and Delta Lake support within the same SparkSession?

Sources:

• https://hudi.apache.org/docs/quick-start-guide
• https://hudi.apache.org/docs/sql_ddl

[torvalds-dev-testbot[bot]]

Abhishek posted:

https://devs-numberlabs.slack.com/archives/C06PDCQPA3W/p1722424798679249?thread_ts=1722425335319449&cid=C06PDCQPA3W

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/235

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/235