fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://fugue-tutorials.readthedocs.io/
Apache License 2.0
1.98k stars 94 forks source link

[FEATURE] Add the namespace concept to Fugue extensions #421

Closed goodwanghan closed 1 year ago

goodwanghan commented 1 year ago

Fugue has very flexible ways to parse custom extensions expressions. We are working with more partners to enable their extensions for Fugue SQL, so there should be a more standard way to host extensions provided by different partners.

We should use a Tuple[str,Any] to represent such extensions, where the first element indicates the domain (provider) of the extension, and the second value can be used to construct the extension by the domain owner.

To enable this in Fugue SQL, we need to enable a new syntax: USING domain:ext_expr, where we use : to separate the domain and extension expression. If domain is provided then Fugue SQL parser should parse this as a tuple (domain, ext_expr) else just a string ext_expr