search

nicolaskruchten / pivottable

Open-source Javascript Pivot Table (aka Pivot Grid, Pivot Chart, Cross-Tab) implementation with drag'n'drop.
https://pivottable.js.org/
MIT License
4.35k stars 1.08k forks source link

CSV weight is about 3,3 Gb #1266

Open luxystech opened 3 years ago

luxystech commented 3 years ago

Hello

Could give us advise how to handle big data process. Our CSV file weights is about 3,3Gb and currently we have 48 Gb RAM, 1 processor with 4 threads and not enough memory to process with PivotUI.

Any recommendations will be very appreciated.

VitaliyMF commented 3 years ago

First of all, pivottable.js is just not suitable for CSVs of this size - as it aggregates the dataset on the client-side (in a web browser). You need to use a solution that performs aggregation on the server-side.

Then, 3.3Gb is too much for quick aggregations simply by processing CSV file directly (even on the server side). It is much better to import CSV data into database; this could be even SQLite - however if you're looking for almost real-time aggregations I may recommend to check single instance of ClickHouse (this is an open source columnar database).