[SUPPORT] Query about executing `hoodie.streamer.transformer.sql` configuration and creating keys using SQL in HoodieStreamer

[torvalds-dev[bot]]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

The user is querying about the execution order of hoodie.streamer.transformer.sql configuration and the possibility of creating keys using SQL in HoodieStreamer.

To Reproduce

Steps to reproduce the behavior:

1. 2. 3. 4.

Expected behavior

A clear and concise description of what the user expects to happen.

Environment Description

• Hudi version :

• Spark version :

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :

• Running on Docker? (yes/no) :

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[torvalds-dev[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev[bot]]

Error generating response

[torvalds-dev[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev[bot]]

Error generating response

[torvalds-dev[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev[bot]]

Error generating response

[torvalds-dev[bot]]

Hello! 👋 Please wait while I search the knowledge base for an answer; this usually takes about 30 seconds

[torvalds-dev[bot]]

Based on the provided documentation snippets:

• Yes, the hoodie.streamer.transformer.sql configuration in org.apache.hudi.utilities.streamer.HoodieStreamer allows for executing an SQL query during write.
• The hoodie.streamer.transformer.sql configuration is specified using the TRANSFORMER_SQL parameter.
• Additionally, it is possible to create a key using SQL and then reference it in hoodie.datasource.write.recordkey.field and hoodie.datasource.write.partitionpath.field in the properties file.

Sources:

• https://hudi.apache.org/docs/configurations
• https://hudi.apache.org/docs/transforms

[torvalds-dev[bot]]

Ticket created successfully. Here is the link to the GitHub issue: https://github.com/torvalds-dev/hudi/issues/60