search

kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
92 stars 89 forks source link

Support ParquetDataSet for polars #284

Closed Sage0614 closed 1 year ago

Sage0614 commented 1 year ago

Context

Parquet is popular storage format, and polars have native support for it, I would like see first class support for eager mode (polars.read_parquet/write_parquet) in the short term and lazy mode (polars.scan_parquent) in the long term

astrojuanlu commented 1 year ago

Hi @Sage0614 ! Could you please try https://github.com/kedro-org/kedro-plugins/pull/170 and let us know if it worked for your use case?

Sage0614 commented 1 year ago

Hi @Sage0614 ! Could you please try #170 and let us know if it worked for your use case?

That looks good to me, one thing to add is the jsonschema should also be updated to reflect the change in #170

https://github.com/kedro-org/kedro/tree/main/static/jsonschema