search

luizperes / simdjson_nodejs

Node.js bindings for the simdjson project: "Parsing gigabytes of JSON per second"
https://arxiv.org/abs/1902.08318
Apache License 2.0
549 stars 25 forks source link

Memory Leak #40

Open ChrisSmith opened 4 years ago

ChrisSmith commented 4 years ago

When running the benchmarks locally, the process gradually consumes all memory on my machine until it runs out of memory (~28GB)

Commenting out simdjson.lazyParse in the benchmark removes leak behavior, so I don't think its the test setup. Printing the node heap statistics shows that it is stable ~80 mb, so the leak is likely in the c++ code.

ChrisSmith commented 4 years ago

isValid and parse don't seem affected, so its specific to lazyParse

luizperes commented 4 years ago

Hi @ChrisSmith, thanks for your input. I will investigate that and get back to you, cheers!

luizperes commented 4 years ago

Hi @ChrisSmith, I took some time to investigate that today and in my machine (2018 Macbook i7 with 16GB RAM) it doesn't leak memory as we would expect, but had a peak of around 5GB but soon released it and had an average usage of 2GB. This on 7 threads as I didn't find a way to control the number of threads from within benchmark lib.

We also have just updated the library. Now it's currently on version 0.5.0, so it could've been a bug on the upstream.

Please also note that simdjson.lazyParse and simdjson.isValid should do basically the same work, although we keep a pointer to the parsed object here. But that is always released here once the record (pointer owner) is removed from the stack.

I will need more info from you, if it still happens. Thanks!

Uzlopak commented 2 years ago

I can also confirm the memory leak.