search

gcarq / rusty-blockparser

Bitcoin Blockchain Parser written in Rust language
GNU General Public License v3.0
370 stars 145 forks source link

"Killed" due to memory run out and poor perormance #28

Closed vizzah closed 4 years ago

vizzah commented 7 years ago

[seems to be a duplicate of #13 - the issue is still actual]

Hi again ;)

I managed to start processing callbacks (and dumping) on the different hardware.

The issue with the program is that it seems to do no memory management and dies as soon as it consumes all available memory and all swap space. Linux process manager sends it a kill signal and it dies with 'Killed' message.

Looking at the performance (for "csvdump"), as soon as memory runs out and swap space is used, the performance degrades considerably (from 600% CPU load it goes to 20% ..etc - the I/O become a bottleneck).

Is there a way to release memory when dumping blocks? Does it have to keep such a large allocation (it got killed after consuming 32 GB RAM and 16 GB swap). When dumping to CSV, why does it need to keep so much data in memory, when what it does is basically doing a data passthru from block files to csv files?

Cheers vZ

ftruzzi commented 6 years ago

Similar issue here? On my system I get:

fatal runtime error: out of memory
Illegal instruction (core dumped)

I have 12GB of RAM, but it looks like it isn't swapping at all. I'm working on the Dogecoin blockchain

jpxue commented 4 years ago

If your issue is: 'memory allocation of 100000000 bytes failed' while its running, try compiling it as 64-bit. Solved the issue for me because the issue was trying to exceed the 4gb memory limit under 32-bit.

In Cargo.toml add:

[build]
target = "stable-x86_64-pc-windows-msvc" #or whatever platform you're targetting

My rust installation only came with the i686-pc-windows-msvc target platform (Windows x86 platform).

You can find the binaries for other platforms at: https://forge.rust-lang.org/infra/other-installation-methods.html Just make you you install them in the right rustup directory: C:\Users\UAC_NAME\.rustup\toolchains\PLATFORM_NAME\

gcarq commented 4 years ago

This should be fixed with the latest release 0.7.0. Feel free to reopen if this is still an issue.