wallet and hard faults

[pocesar]

I noticed every time I open rai_wallet, the audio from spotify starts slowing down or skipping / glitching. loaded windows 10 performance monitor, and it seems that the wallet is generating a lot of hard faults per sec.

also, the I/O that is using my SSD is pretty high, is there a way to move the data.ldb somewhere else?

[thehen101]

The audio skipping may be because you have a CPU with a low thread count - consequently the RaiBlocks wallet and Spotify both execute on the same thread so when one demands a lot of processing power the other program may lag behind. It could also be the thread waiting for IO. As for the data path, try launching the wallet with the data_path command (I haven't tested it, try it out yourself): https://github.com/clemahieu/raiblocks/wiki/Command-line-interface#--data_pathpath

Note that this Raiblocks wallet does do a lot of IO, so perhaps storing the blockchain on an SSD isn't a great idea if that's what you were planning.

[pocesar]

I mean, it's currently using my SSD (because it's where my windows is installed, therefore the AppData is there as well), and I moved it to my HDD. but I solved it using a hard link, the only thing that caught my eye is the number of hard faults

the CPU was pretty low, and indeed it's an old CPU (Core 2 Quad), but it doesn't happen (sound glitching / slow downs) even mining some CPU intensive coins like cryptonight

[ragardner]

Audio skipping was also happening for me during syncing, not sure about when not syncing, the audio was being played over youtube running on chrome, no spotify running, with version 8.0 of the wallet on windows 10 64, have not tried latest release candidate yet

[nullbio]

I'm having this problem as well (audio and mouse glitching) during syncing. Particularly noticeable on Spotify and Discord. Using a relatively new i7 6700k, noticed CPU usage around 30% on Windows 10 64.

[vereten]

I`m experiencing same issue on Debian/Testing Linux distribution. RaiBlocks wallet is COMPLETELY freezing whole operating system in some intervals for few seconds.

[jbe]

I have a studio sound card (orion 32), which the wallet effectively ddos-es (i have to close the wallet if i want audio). Other than this, resource use is not a problem. This is likely connected with the high number of hard faults.

Although i am not familiar with lmdb, my guess is that it uses memory-mapped files. The bursts of random access reads and writes are probably eating up all the operating system io resources that the audio driver also needs to maintain sync.

I am not exactly sure what benefits lmdb provides, my gut feeling is that it could be replaced with a custom data model in C/C++ that matches our specific requirements, which would improve performance and eliminate problems like this.

For instance, instead of immediately verifying new blocks as they come in (random access), pending verification operations could be kept in a queue (backlog), and be processed in a more sorted/sequential manner while sequentially iterating over existing blocks (this might be much cache-friendlier?). Implementing a good custom caching heuristic for blocks and keeping a few gb in memory during intensive operations like syncing would probably also help a lot..? Also, append-only disk persistence?

A lot that could be done, but none of it trivial, and it would probably require the full time attention of at least one person for some time to work out.

I could be wrong though, maybe its something else.

[cryptocode]

@jbe lmdb is used with success in many crypto projects (including monero), and it's considered one of of the fasted and most reliable KV stores out there with key-ordering support.

So maybe we need to look at how lmdb is used, but I don't think lmdb itself is to blame.

Also, append-only disk persistence?

While lmdb has immutable pages, it does recycle pages from a freelist.

[jbe]

@cryptocode I'm not trying to blame lmdb, but thanks for clarifying some of the benefits.

One significant difference between raiblocks and Monero is that Monero doesn't need to do a lot of small random access writes. "Traditional" blockchains have larger blocks, written sequentially, and an absolute ordering of blocks. Rablocks has a huge amount of very small blocks that are only partially ordered, with writes happening randomly all across the block space. This is a different access pattern, with different optimization considerations.

When millions of small blocks are accessed randomly though a disk kv-store, it will cause lots of page faults, beacuse the OS typically has to swap in a new page for every single block that is retrieved or written. As you mentioned, this is not really about lmdb, but how it is being used.

I'm not in a position to say how exactly to solve this (only throwing out some rough thoughts), but i believe it is an issue that the wallet needs to handle in some way or other (it will only get worse).

[pocesar]

interesting experiment (with graphs) https://medium.com/@bnp117/stress-testing-the-raiblocks-network-568be62fdf6d (peaked at 44Mbps upstream) and was precomputed for hours using a GPU. HD I/O peaked at 737 MB/s, averaging 240 MB/s

[jbe]

For those interested, #540 is probably strongly related to this, and contains some nice research. It also points out that lmdb is indeed a memory mapped b-tree (that we are accessing randomly), which explains the high rate of page faults.

[rkeene]

There is an on-going effort to improve node disk utilization, network utilization, and CPU utilization. One such thing that may help with page faults is that we are going to split out the database backend into a more modular system that will support more database engines in addition to LMDB. I'm going to close this issue so we can track it as part of the effort to split the database backend.

Duplicate of #1093