[Feature] Support polars in your ParquetDataNode (or maybe some sort of dataset abstraction?)

kszlim commented 4 months ago

Description

It looks like ParquetDataNode assumes you're using pandas, it would be nice if it supported Polars too? Furthermore, I think it'd be useful to support ParquetDatasets, where you have a hive partitioned directory (that could potentially live in the cloud) and have that as a node instead.

A common use case (for me) in polars is to query and process some data in s3. It would be nice to encode this in taipy too.

Solution Proposed

Not sure how it might look.

Impact of Solution

Would give first class support to polars.

Additional Context

No response

Acceptance Criteria

[X] Ensure new code is unit tested, and check code coverage is at least 90%.
[X] Create related issue in taipy-doc for documentation and Release Notes.
[X] Check if a new demo could be provided based on this, or if legacy demos could be benefit from it.
[X] Ensure any change is well documented.

Code of Conduct

[X] I have checked the existing issues.
[ ] I am willing to work on this issue (optional)

jrobinAV commented 4 months ago

We like the idea. We need to work on better defining the use case and the requirements.

Thank you very much.

jrobinAV commented 3 months ago

Depends on the output of Avaiga/taipy-enterprise#411 .

jrobinAV commented 3 weeks ago

This feature is implemented in Taipy Enterprise Edition 4.0.

Avaiga / taipy