risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.03k stars 577 forks source link

Use UDFs as sources #16223

Closed bakjos closed 1 month ago

bakjos commented 7 months ago

Background

With the amount of sources and how different is consumed by the clients, using UDFs could help to support more sources:

Design

The UDFs are already supporting tables, and use yield to generate the values through streaming futures https://github.com/risingwavelabs/risingwave/blob/ed75502880af4fad364b9821ab85e7026188063d/src/expr/core/src/table_function/user_defined.rs#L93-L132.

This functionality could be translated to tables/sources to emit records when a new value is received from the UDF, using a sintax like CREATE [TABLE|SOURCE ] .. AS <function_name> (param1, param2)

Future Optimizations

Discussions

No response

Q&A

No response

github-actions[bot] commented 3 months ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄

stdrc commented 1 month ago

https://github.com/risingwavelabs/rfcs/pull/87#issuecomment-2333169493