User Defined Functions - Support custom binaries with STDIN/STDOUT

[danthegoodman1]

It would be extremely useful if we could create our own user defined functions in any language we choose, adding unparalleled flexibility to CRDB. It would provide capabilities like self-run version of Cloudflare's Durable Objects.

Clickhouse does this very well with strong specifications and error handling policies: https://clickhouse.com/docs/en/sql-reference/functions/

Processing through STDIN & STDOUT allows any lanugage to be used as long as the function respects the expect data formats.

Support for SELECTs would be more than enough, as the result of a SELECT could be inserted. For example one function I've always wanted in CRDB is HyperLogLog support. Functions could be made like hll_merge(IDENTIFIER, NEW_VAL), where to update a HLL you UPSERT the result of hll_merge, then use a function hll_count(IDENTIFIER) to read it.

Some immediate capabilities that could be unlocked are:

1. HyperLogLogs
2. Bloom filters
3. Complex running aggregations (materialized as well)
4. Room/multiplayer session management (e.g. collaborating on a google doc)

Jira issue: CRDB-22319

[blathers-crl[bot]]

Hello, I am Blathers. I am here to help you get the issue triaged.

It looks like you have not filled out the issue in the format of any of our templates. To best assist you, we advise you to use one of these templates.

I was unable to automatically find someone to ping.

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[danthegoodman1]

Another thought: Since we don't have access to the raw nodes on CRDB Dedicated or Serverless, might be great to have it pull the function binary/file from S3.

Not sure the implications for serverless (not supported, or ad-hoc pull?)

(We are Dedicated users so that's where we need it :P)

[mgartner]

Thanks for the report! We'd love to support UDFs, triggers, and stored procedures written in more languages than just SQL and PL/pgSQL. We've discussed supporting WASM functions as a way to provide more flexibility to users. But we're still a ways away from that.

[danthegoodman1]

Yeah wasm is probably pretty good, but is a tad painful for Python, js, and golang since they don’t have native implementations

The wasmedge runtime for js is pretty ok but nothing really comes close to Rust in terms of that

even just v8 (or another runtime) support could be nice