search

penberg / helix

Helix, a market data feed handler for C and C++.
Other
110 stars 35 forks source link

Memory Usage #48

Open nickos556 opened 6 years ago

nickos556 commented 6 years ago

Great tool, thanks so much for sharing! I am running helix-trace using the following command line:

./helix-trace -i /media/sf_data/ITCH5.0/08302017.NASDAQ_ITCH50 -s AAPL -P nasdaq-binaryfile-itch50 -f csv -o AAPL.csv

08302017.NASDAQ_ITCH50 is downloaded from ftp://emi.nasdaq.com/ITCH/ and is roughly 5.4gb

When I run the tool, it steadily uses more and more memory, until it crashes (I only have 4gb of memory). I am running using Fedora 26. Any ideas what is going on? Does helix not delete orders after they are executed or cancelled?

thanks Nick

penberg commented 6 years ago

@nickos556 I'm able to process a similarly sized file, but I also have 32 GB of RAM. Could you please try reproducing with a debug build that enables AddressSanitizer?

mkdir -p build/debug
cd build/debug
cmake -DCMAKE_BUILD_TYPE=debug ../..
penberg commented 6 years ago

@nickos556 I ran helix-trace using Massif and the peak memory usage is around 2.8 MiB for a 7.1 GiB ITCH 5.0 file. The memory usage increase you're seeing is an artifact of helix-trace using mmap() for accessing the file but the kernel will manage that memory for you and shouldn't cause you to run out of memory.