[SUPPORT] How to store schema in schema registry for capturing nested arrays and JSONs when onboarding data from Mongo source for Hudi table?

[torvalds-dev-testbot[bot]]

Tips before filing an issue

• Have you gone through our FAQs?
• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

I am looking for suggestions on how to store the schema in the schema registry to capture nested arrays and JSONs with column names for Hudi table onboarding when the source is a Mongo data source, considering a static schema nature of data.

To Reproduce

Expected behavior

Environment Description

• Hudi version :
• Spark version :
• Hive version :
• Hadoop version :
• Storage (HDFS/S3/GCS..) :
• Running on Docker? (yes/no) :

Additional context

Stacktrace

[torvalds-dev-testbot[bot]]

Relevant Sources:

https://hudi.apache.org/docs/faq_storage

https://hudi.apache.org/docs/schema_evolution

https://aws.amazon.com/blogs/big-data/automate-schema-evolution-at-scale-with-apache-hudi-in-aws-glue/

https://github.com/apache/hudi/issues/10895

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

Hudi supports schema reconciliation, allowing you to apply the target table schema on incoming data to handle missing fields by injecting null values. Schema evolution is also supported on write, allowing actions like adding nullable fields or promoting field datatypes. To store complex nested arrays and JSONs with column names for Hudi table onboarding using a static schema nature from a Mongo data source, you can follow the schema evolution best practices and adjust the schema accordingly.

Follow up questions:

• Do you have any specific requirements for the schema evolution process?
• Are there any additional constraints or considerations for the schema design?
• Can you provide more details about the structure of the data from the Mongo data source for schema adaptation?

Sources:

• https://hudi.apache.org/docs/faq_storage
• https://hudi.apache.org/docs/schema_evolution

[torvalds-dev-testbot[bot]]

Please connect your Zendesk account before creating a ticket!

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/249