search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Disable warning that parquet table cache cannot dematerialize sa.Table #109

Open windiana42 opened 11 months ago

windiana42 commented 11 months ago

The warning that dematerialization may be inhibited by "conditions" such as missing installed packages is great! But for the parquet table cache, there should be a list of input_types for which it is obvious that they cannot be parquet cached.

Maybe the hook could return different responses for "I don't know this type" and for "I know this type, and I know that it is not suited for local table caching". Other solution options are possible including a hand curated match list of ill-suited types for local table cache.

IvanDimitrovQC commented 5 months ago

I can pick this one up - it will help me familiarize myself with the code and since it's just error/warning message related it's something for which I don't really need to worry too much about breaking.