search

javascriptdata / danfojs

Danfo.js is an open source, JavaScript library providing high performance, intuitive, and easy to use data structures for manipulating and processing structured data.
https://danfo.jsdata.org/
MIT License
4.81k stars 209 forks source link

Memory heap limit exceeded #84

Closed reevr closed 3 years ago

reevr commented 3 years ago

When I try to load a csv with 1 million rows, the heap memory gets exceeded. I know we need to increase the MAX_OLD_SPACE_SIZE before running the application, but what I see is, DanfoJs is consuming more memory than pandas. Memory gets exhausted fast with few thousands of rows getting loaded I hope this gets fixed soon, as DanfoJs is a promising future to NodeJs data processing capabilities.

steveoni commented 3 years ago

good pointer, we will surely work on this. And you are also free to help out.

risenW commented 3 years ago

When I try to load a csv with 1 million rows, the heap memory gets exceeded. I know we need to increase the MAX_OLD_SPACE_SIZE before running the application, but what I see is, DanfoJs is consuming more memory than pandas. Memory gets exhausted fast with few thousands of rows getting loaded I hope this gets fixed soon, as DanfoJs is a promising future to NodeJs data processing capabilities.

I understand your plight. I'm looking towards streaming operations and parallelization of most of the core operations for danfo. This will help solve this issue.

github-actions[bot] commented 3 years ago

Stale issue message