[SUPPORT] Typical data transformation stages from raw data to gold data when using DeltaStreamer in a data pipeline

[torvalds-dev-testbot[bot]]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

1. 2. 3. 4.

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :

• Spark version :

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :

• Running on Docker? (yes/no) :

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[torvalds-dev-testbot[bot]]

Relevant Sources:

https://www.linkedin.com/pulse/hudi-streamer-delta-hands-on-guide-local-ingestion-from-soumil-shah-jssse/?utm_source=share&utm_medium=member_ios&utm_campaign=share_via

https://bigdataboutique.com/blog/introduction-to-apache-hudi-c83367

https://www.onehouse.ai/blog/how-apache-hudi-simplifies-mpp-data-warehouse-migrations

https://apache-hudi.slack.com/archives/C4D716NPQ/p1711400241290649

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

In a typical data transformation pipeline using DeltaStreamer, the stages include reading Avro messages from Kafka with Schema Registry, deserializing them using the schemas, and ingesting the data into a Hudi-based data lake. Delta Streamer serves as the component for efficient and continuous data ingestion, ensuring standardized data format across the pipeline.

---

Follow up questions:

• Could you provide more information on the specific configurations used for consuming Avro messages from Kafka in the Delta Streamer setup?
• Are there any particular considerations or best practices to keep in mind when configuring the Schema Registry for use within the data transformation pipeline?
• Can you elaborate on the benefits or advantages of leveraging DeltaStreamer for incremental ETL tasks in comparison to traditional batch processing workflows?

Sources:

• https://www.linkedin.com/pulse/real-time-data-processing-postgres-debezium-kafka-schema-soumil-shah-li19e/?utm_source=share&utm_medium=member_ios&utm_campaign=share_via
• https://hudi.apache.org/blog/2020/08/20/efficient-migration-of-large-parquet-tables/#spark-data-source
• https://www.linkedin.com/pulse/from-datalake-microservices-unleashing-power-apache-hudis-soumil-shah-ylkoe/?trk=article-ssr-frontend-pulse_more-articles_related-content-card

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/251