search

fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://fugue-tutorials.readthedocs.io/
Apache License 2.0
1.92k stars 94 forks source link

[FEATURE] AnyDataFrame should be recognized by Creator, Processor and Ouputter #492

Closed goodwanghan closed 11 months ago

goodwanghan commented 11 months ago

This should be a straightforward change, we need this to work:

from fugue import AnyDataFrame

def my_processor(df:AnyDataFrame) -> AnyDataFrame:
    return df
PROCESS USING my_processor

With this change, or functions following fugue api conventions will be able to be used as fugue extensions.