search

holgerbrandl / krangl

krangl is a {K}otlin DSL for data w{rangl}ing
MIT License
560 stars 50 forks source link

Reading "big" csv files #142

Open nlhnt opened 2 years ago

nlhnt commented 2 years ago

Hello, I increased the heapsize to 16G for my kernel with krangl, but reading a CSV file which has 7 columns (int64, str, str, str, int64, str, str) and about 4*e6 rows (almost 800M with utf8 encoding) didn't quite work.
I am stuck with heap size error, Python's pandas was able to load it without much trouble. Unfortunately I face a task where I have to iterate through this table row by row, and Python's loops are not quite useful here (I mean they work, but it not takes a couple of hours to work through that).
Julia's DataFrame.jl was able to load this frame into memory as well (it really is not that big, takes around 2G of RAM on a Windows machine).

holgerbrandl commented 2 years ago

This is indeed a problem with the underlying current implementation which is known to not scale well.

@nikitinas Did you come up with a more memory-efficient way to read big tables in https://github.com/Kotlin/dataframe

Soldalma commented 2 years ago

I have the same problem with a dataframe of 25000 rows and 700 columns. Actually even after I delete most of the rows the problem persists.