Feature request: parquet data files support

finos / perspective

A data visualization and analytics component, especially well-suited for large and/or streaming datasets.

https://perspective.finos.org/

Apache License 2.0

8.57k stars 1.19k forks source link

Feature request: parquet data files support #342

Closed RandomFractals closed 1 year ago

RandomFractals commented 5 years ago

how hard would it be to add parquet data files support to the perspective viewer?

in relation to this vsscode ext. I've decided to work on utilizing your lib:

https://github.com/RandomFractals/vscode-data-preview

I belive it can be done in JS with this lib I found:

https://github.com/ZJONSSON/parquetjs

most likely just a matter of providing proper data read interfaces for perspective viewer to consume and display that data

mhkeller commented 1 year ago

I'm also curious if perspective could be pointed to a parquet file. I was looking into using duckdb to export database queries as parquet files and then pointing perspective at that file to load the results.

texodus commented 1 year ago

This feature is out-of-scope for Perspective. For the Python side, pyarrow and friends already support this (we would just recompile this support into Perspective anyway). For JavaScript, Perspective is designed to be used as a component, not a platform, preferring small size and performance to coming bundled with every possible format.

If you're looking for a turn-key platform approach to BI that incorporates a wide variety of data adapters, analytics, persistence and discovery features, check out our enterprise platform.

mhkeller commented 8 months ago

For anyone looking to try this, here's a script that will read in a parquet file and give you back the buffer representation of an arrow file: https://gist.github.com/mhkeller/855ca5c0a6582e4ead7d36e6f8169fdd

There is likely a better way to do this and I'm not taking this approach, but sharing this here in case it helps someone.