Support larger databases

gristlabs / grist-core

Grist is the evolution of spreadsheets.

https://www.getgrist.com/

Apache License 2.0

6.55k stars 286 forks source link

Support larger databases #43

Open paulfitz opened 2 years ago

paulfitz commented 2 years ago

Some UI features are only practical for medium-sized databases. We could do a better job of supporting essential operations on larger documents.

fulldecent commented 3 months ago

A great test for addressing this issue is at:

https://github.com/gristlabs/grist-core/issues/901#issuecomment-2008651235

And our required performance metrics might be:

Use a Docker container with 512 MB RAB
Use web UI upload to load a 1 GB CSV database
Use 10 simultaneous visitors to view random pages of that database

rswgnu commented 2 months ago

Excel has a limit of 1 million rows in a sheet. Business users are processing much larger datasets these days. So one motivation to use your tool would be efficient loading and processing of larger datasets, maybe using Polars under the hood.