[SUPPORT] Handling multiple DeltaStreamers writing to the same target table

[torvalds-dev-testbot[bot]]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced A clear and concise description of the problem.

To Reproduce Steps to reproduce the behavior: 1. 2. 3. 4.

Expected behavior A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :
• Spark version :
• Hive version :
• Hadoop version :
• Storage (HDFS/S3/GCS..) :
• Running on Docker? (yes/no) :

Additional context Add any other context about the problem here.

Stacktrace Add the stacktrace of the error.

[torvalds-dev-testbot[bot]]

Relevant Sources:

https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6

https://hudi.apache.org/docs/faq_writing_tables

https://hudi.apache.org/docs/concurrency_control

https://github.com/apache/hudi/issues/10246

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

Follow-up questions:

1. Can you describe the version of Hudi and other tech stack in your environment? If you are facing an error, please provide a full stack trace and describe what operations led to the errors.
2. What is the indexing scheme you are using, and what table type are you utilizing for Hudi?
3. Are there any specific configurations you have set for the Hudi deltastreamers writing to the same target table?
4. How do you manage your data writer processes for the same Hudi table?
5. Could you share your Hudi timeline to understand the sequence of operations by multiple deltastreamers writing to the table?

Sources:

• https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6
• https://hudi.apache.org/docs/concurrency_control

[torvalds-dev-testbot[bot]]

Follow-up questions:

1. Can you describe the version of Hudi and other tech stack in your environment? If you are facing an error, please provide a full stack trace and describe what operations led to the errors.
2. What is the indexing scheme you are using, and what table type are you utilizing for Hudi?
3. Are there any specific configurations you have set for the Hudi deltastreamers writing to the same target table?
4. How do you manage your data writer processes for the same Hudi table?
5. Could you share your Hudi timeline to understand the sequence of operations by multiple deltastreamers writing to the table?

Sources:

• https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6
• https://hudi.apache.org/docs/concurrency_control

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I generate the follow up questions; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

• Hudi supports multi-writer capabilities, allowing multiple writers to concurrently write to the same table as long as they are updating different sets of data. This feature is crucial for enhancing system throughput and managing data ingestion from various sources.
• Concurrent writes are managed using Optimistic Concurrency Control (OCC), where each writer proceeds optimistically assuming success and resolves conflicts before committing to ensure only one writer succeeds when updating the same data. For example, when two writers concurrently update overlapping file groups, only the first one completing the update will succeed.
• Hudi provides a Conflict Resolution component to handle conflicting operations and employs strategies to resolve conflicts. Default strategy SimpleConcurrentFileWritesConflictResolutionStrategy operates at the File Group level.
• If two concurrent writers update data from different file groups, both writes will succeed. But if they update overlapping file groups, conflicts are resolved accordingly to either commit or abort the transaction.
• In scenarios where concurrent writers interact with the same partition and there is no guarantee of non-overlapping writes, write failures can occur. However, retries are likely to succeed once previous conflicts are resolved.

Sources:

1. https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6
2. https://hudi.apache.org/docs/concurrency_control

[torvalds-dev-testbot[bot]]

• Hudi supports multi-writer capabilities, allowing multiple writers to concurrently write to the same table as long as they are updating different sets of data. This feature is crucial for enhancing system throughput and managing data ingestion from various sources.
• Concurrent writes are managed using Optimistic Concurrency Control (OCC), where each writer proceeds optimistically assuming success and resolves conflicts before committing to ensure only one writer succeeds when updating the same data. For example, when two writers concurrently update overlapping file groups, only the first one completing the update will succeed.
• Hudi provides a Conflict Resolution component to handle conflicting operations and employs strategies to resolve conflicts. Default strategy SimpleConcurrentFileWritesConflictResolutionStrategy operates at the File Group level.
• If two concurrent writers update data from different file groups, both writes will succeed. But if they update overlapping file groups, conflicts are resolved accordingly to either commit or abort the transaction.
• In scenarios where concurrent writers interact with the same partition and there is no guarantee of non-overlapping writes, write failures can occur. However, retries are likely to succeed once previous conflicts are resolved.

Sources:

1. https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6
2. https://hudi.apache.org/docs/concurrency_control

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/248

[torvalds-dev-testbot[bot]]

Hello! :wave: Please wait while I generate the follow up questions; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

Based on the documentation snippets:

• Apache Hudi has multi-writer support, allowing two writers to concurrently write to Hudi and commit as long as they update different sets of data without causing table corruption (https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6>).

• Hudi employs Optimistic Concurrency Control (OCC) where conflicting writes to the same data will result in one succeeding and the other aborting (https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6).

• Concurrent writers updating overlapping data may lead to conflicts, with only one succeeding while another aborts (https://medium.com/@simpsons/multi-writer-support-with-apache-hudi-e1b75dca29e6).

• Multi-writing through Hudi Streamer requires adding specific configurations for optimistic concurrency control in the properties file (<http://httpshudi.apache.org/docs/concurrency_control).

Based on the above information, if two DeltaStreamers are running for the same target table and attempt to concurrently write to overlapping file groups, one of them will succeed while the other will fail due to conflict resolution mechanisms in place. Hence, this won't result in table corruption, but instead, conflicting operations will be handled, ensuring the integrity of the data being written. :large_blue_square: Confidence Score: 93.46% :loudspeaker: Publish Answer button >torvalds.dev is learning and improving. React below to provide feedback! :+1: 0 button :-1: 0 button Submit button, with interactive elements