[SUPPORT] Issue with row level deletion using write operation delete

[torvalds-dev-testbot[bot]]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

I am currently trying to perform row level deletion using the write operation delete in one of my tables (COW) with Hudi version 0.14.0. Despite using the provided configuration for deletion, no rows are getting deleted when I execute my PySpark job. I verified the rows before deletion to ensure correct filtering.

To Reproduce

Steps to reproduce the behavior:

1. Execute PySpark job for row level deletion.

Expected behavior

I expected the specified rows to be deleted as per the filter condition.

Environment Description

• Hudi version : 0.14.0
• Spark version :
• Hive version :
• Hadoop version :
• Storage (HDFS/S3/GCS..) :
• Running on Docker? (yes/no) :

Additional context

I am not encountering any errors during the process, just the rows are not being deleted as expected.

Stacktrace

Add the stacktrace of the error.

[torvalds-dev-testbot[bot]]

Relevant Sources:

https://www.linkedin.com/pulse/small-talk-apache-hudi-ashok-kumar-kunkala-3ldge/

https://hudi.apache.org/docs/quick-start-guide

https://datacouch.medium.com/getting-started-with-apache-hudi-711b89c107aa

https://apache-hudi.slack.com/archives/C4D716NPQ/p1713562820086629

[torvalds-dev-testbot[bot]]

Hello! 👋 Please wait while I generate the follow up questions; this usually takes about 30 seconds

[torvalds-dev-testbot[bot]]

Based on the provided documentation snippets, it seems like the row level deletion with the Hoodie write operation "delete" might not be working as expected in your scenario. Here are some insights and steps to potentially debug this issue:

• The Delete flow in Hoodie is a special case of the Upsert flow, where HoodieKeys are transformed during the "transform input" step to identify the records to delete.
• It's crucial to ensure that the source dataframe contains Hudi's meta fields or a key field is configured for the deletes to work properly.
• The Save mode should be set to Append for the deletion operation.
• The operation is a hard delete, meaning that the target records will not exist in the new FileSlices of the corresponding FileGroups.
• Make sure to check the conditions during the filtering process to guarantee that the rows intended for deletion are being correctly identified.

If you are still facing issues with the row level deletion despite following these guidelines, further investigation into the filtering conditions and the underlying data might be necessary to resolve the problem effectively.

Sources:

• https://hudi.apache.org/docs/write_operations
• https://blog.datumagic.com/p/apache-hudi-from-zero-to-one-310
• https://hudi.apache.org/docs/quick-start-guide
• https://datacouch.medium.com/getting-started-with-apache-hudi-711b89c107aa

[torvalds-dev-testbot[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/numberlabs-developers/hudi/issues/250