Open j-bennet opened 1 year ago
Good catch @j-bennet. I think adding dataset={"schema": dt.schema().to_pyarrow()}
as a keyword to this read_parquet
call
should do the trick. Though it'd be nice if someone could confirm this is the case.
I think adding
dataset={"schema": dt.schema().to_pyarrow()}
as a keyword to thisread_parquet
call should do the trick. Though it'd be nice if someone could confirm this is the case.
I think Delta Log also contains columns stats, so maybe we can avoid gathering those.
In dask-deltatable, when calling dd.read_parquet, perhaps we can reuse the metadata already preserved in delta json, instead of collecting it from parquet files all over again.
Here:
https://github.com/dask-contrib/dask-deltatable/blob/cd731a9a237c84c824aff1e8ea61d4ba6988f3b9/dask_deltatable/core.py#L196
It looks like
dd.read_parquet
will have to go through the parquet files to read the metadata, but theDeltaTable
should have all that info already.