search

Eugene-Mark / bigdata-file-viewer

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.
GNU General Public License v2.0
282 stars 54 forks source link

Add pagination, do not load everything into memory #20

Open denisovlev opened 3 years ago

denisovlev commented 3 years ago

I was looking at the code and it seems that all the records from the file are loaded into memory. Some files can be quite large. Perhaps adding pagination and not loading everything into memory would be better?

Eugene-Mark commented 3 years ago

@denisovlev Yes for sure, it's in the to-do list. It's welcomed if you're interested to contribute on it.

Eugene-Mark commented 10 months ago

Close it since it's over years, will reopen it if the feature is in the roadmap.