search

fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://fugue-tutorials.readthedocs.io/
Apache License 2.0
1.98k stars 94 forks source link

[BUG] pa.Table as transformer output throws an exception #490

Closed goodwanghan closed 1 year ago

goodwanghan commented 1 year ago

Minimal Code To Reproduce

import pyarrow as pa
import fugue.api as fa

import pandas as pd

df = pd.DataFrame([[0]], columns=["a"])

def tr(df:pa.Table) -> pa.Table:
    return df

fa.transform(df, tr, schema="*")

Describe the bug The problem is on _PyArrowParam, the to_output_df needs to handle schema, also we need to more test coverage on this