fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://fugue-tutorials.readthedocs.io/
Apache License 2.0
1.98k stars 94 forks source link

[BUG] Fix compatibility issues for duckdb 0.8.0+ #471

Closed goodwanghan closed 1 year ago

goodwanghan commented 1 year ago

In Duckdb 0.8,

  1. DuckDBPyTypes are created to replace the simple string representation. This breaks the code using DuckDBPyRelation.types. So we need to make Fugue work for both duckdb<0.8.0 and >=0.8.0
  2. Duckdb starts to use arrow large_string, but Fugue doesn't directly support this type.
goodwanghan commented 1 year ago

@bitsofinfo