Closed xingyaoww closed 2 years ago
cool work.
Besides, this teach me that I should use max_depth=10
when init a feedhandler
I run the same script on a machine with larger RAM, those huge spikes seem to go away. I also increase the RAM of my AWS instance to 1G and try again -- now the latency is within a reasonable range (< 70ms spike for order book with depth 16320+11022=27342).
The result kind of confirmed this issue may be caused by potential memory allocation/page fault (though I think it needs more experiments to actually make the claim).
Also I noticed we can get some decent performance improvement (almost 10x) when switching from orderbook.book.bids.index(i)
to iterates the first 20 elements (i.e. price) from orderbook.book.bids.keys()
, then do orderbook.book.bids[idx]
to fetch the volume. Although I'm not sure why: seems they both are some Py*_GetItem
operations (.index
use 2 GetItem to fetch key and value separately), and .keys()
even need to return a larger slice of keys -- why it is even faster though?
Describe the bug The latency of processing the order book (e.g., get the first 20 levels for bid and ask) increases as time goes by (especially when the total levels of the order book increase -- similar to #406). But the more problematic issue would be that the latency spikes every few iterations when book levels increase to a certain level (e.g., latency increases to ~350ms around 12,500 depth).
To isolate whether it is caused by some os scheduling problem that swap the process out of CPU, I use
isolcpus
andtaskset
(see here) to make sure this is the only one processes that is running on a CPU.Could it be something related to memory allocation on the orderbook's C implementation (as orderbook grows)? Note that I'm running this test in a small AWS EC2 instance where resources (RAM) are limited (~500M), and I'm not sure if this is sufficient for cryptofeed to process the orderbook (I can also run it on another machine with larger memory to double-check).
Any help on this would be appreciated! I'll be happy to PR when possible.
To Reproduce
Redirect the stdout to a log file and run the following scripts for a few hours.
Then use the script to parse the log file and plot:
Expected behavior It is expected to cost additional time when the depth levels grow (hopefully latency increases slower), but those spikes shouldn't be there.
Screenshots See bug description.
Operating System: Linux (Ubuntu 20.04, linux-image-5.15.0-1015-aws)
Cryptofeed Version
cryptofeed
version 2.2.3, installed from pypi.