[SUPPORT] Query about using Hudi in Pyspark for BigQuery sync

[torvalds-dev-testbot[bot]]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

The user is inquiring about the support for using Hudi in Pyspark for syncing with BigQuery. They are unable to find any resources on how to implement this.

To Reproduce

Steps to reproduce the behavior:

1. 2. 3. 4.

Expected behavior

A clear and concise description of what the user expected to happen.

Environment Description

• Hudi version :

• Spark version :

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :

• Running on Docker? (yes/no) :

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[torvalds-dev-testbot[bot]]

Relevant Sources:

https://dzone.com/articles/delta-hudi-and-iceberg-the-data-lakehouse-trifecta

https://hudi.apache.org/docs/sql_queries

https://hudi.apache.org/docs/gcp_bigquery

https://hudi.apache.org/docs/syncing_metastore

https://github.com/apache/hudi/issues/7117

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/225

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/225

[torvalds-dev-testbot[bot]]

Hello! :wave: Please wait while I generate the follow up questions; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

Follow up questions:

1. Can you describe the version of Hudi and other tech stack in your environment?
2. What specific Hudi write configurations are you using for BigQuery sync with Pyspark code?
3. Could you provide the Pyspark code snippet you are currently using for implementing Hudi in BigQuery sync?

Sources:

• https://hudi.apache.org/docs/sql_queries
• https://hudi.apache.org/docs/gcp_bigquery
• https://datacouch.medium.com/getting-started-with-apache-hudi-711b89c107aa :large_blue_square: Confidence Score: 99.83% :loudspeaker: Publish Answer button >torvalds.dev is learning and improving. React below to provide feedback! :+1: 0 button :-1: 0 button

[torvalds-dev-testbot[bot]]

Please connect your slack account on torvalds.dev to publish the message on behalf of the user and try again later. You can accept permission at - https://torvalds-dev-nl-ghco-nl-ghcos-projects.vercel.app/settings

[torvalds-dev-testbot[bot]]

Please connect your slack account on torvalds.dev to publish the message on behalf of the user and try again later. You can accept permission at - https://torvalds-dev-nl-ghco-nl-ghcos-projects.vercel.app/settings