Closed soumilshah1995 closed 6 months ago
@soumilshah1995 Looks like you missed to enable the record index.
I was able to run successfully below command by just adding --hoodie-conf "hoodie.metadata.record.index.enable=true" \
-
~/plain_spark/spark-3.2.4-bin-hadoop3.2/bin/spark-submit \
--class org.apache.hudi.utilities.HoodieIndexer \
--packages 'org.apache.hudi:hudi-spark3.2-bundle_2.12:0.14.1' \
--master 'local[*]' \
--executor-memory 1g \
~/jars/0.14.1/spark32/hudi-utilities-slim-bundle_2.12-0.14.1.jar \
--mode scheduleAndExecute \
--base-path file:///tmp/hudi_cow_read \
--table-name hudi_table \
--index-types RECORD_INDEX \
--hoodie-conf "hoodie.metadata.enable=true" \
--hoodie-conf "hoodie.write.concurrency.mode=optimistic_concurrency_control" \
--hoodie-conf "hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider" \
--hoodie-conf "hoodie.metadata.record.index.enable=true" \
--hoodie-conf "hoodie.metadata.index.async=true" \
--parallelism 2 \
--spark-memory 2g
let me try this :D
That approach does indeed work.
I have a basic question: If you wish to enable record-level indexing (RLi) in offline mode but disable it in regular jobs, how would we go about it? Would there be a flag to deactivate RLi during writing, allowing us to execute it asynchronously?
@soumilshah1995 If we want to do that, we can disable hoodie.metadata.record.index.enable in the ingestion, so it doesn't use RLI during regular ingestion.
Interesting Thanks
hey @ad1happy2go @codope : looks like there is some mis understanging on how to use async indexer. when enabling async indexer to build say RLI, ingestion also should have async indexing enable for RLI. we can't completely disable from regular ingestion job. Can you folks follow up on any doc enhancements. CC @soumilshah1995
Yes that is correct. It's mentioned as a limitation in https://hudi.apache.org/docs/metadata_indexing#caveats
Thanks for heads up guys
Greetings,
I hope this message finds you well. There was a recent discussion in the group regarding the possibility of changing the index type from Bloom to RLI on an older table. My understanding was that RLI needed to be created on a fresh table. However, there have been discussions on Slack suggesting that we can utilize the Hudi indexer for this purpose.
Before proceeding, I would like to kindly request clarification and verification on whether such a transition is feasible or not. Your insights on this matter would be greatly appreciated.
Error
Thank you for your attention to this inquiry.