Sync speed bottlenecks

[noaho]

As observed yesterday, CPU utilization while syncing is fairly slow, and re-sync takes quite some time.

I would like to run some testing to determine the current speed bottlenecks and see if we can improve in some way.

TODO:

• Benchmark current sync time
• See if it's IO/Network/CPU/Memory/USB bound
• Look for further improvements

My hypothesis is that it's write bandwidth of the USB stick holding us up, and if we optimize the volatile volume to have a small write-back cache it will improve the USB stick's performance substantially. Maybe even enough that we can consider using a slower raspberry Pi. https://whatis.techtarget.com/definition/write-back

See for example this video which shows how badly fast a memory card performs when it's not being used the way it expects. https://www.youtube.com/watch?v=BrcBKpE9aQg

To be clear I am only proposing this for the volatile USB, we don't want this for important.

And this is only a hypothesis open to be tested. I've ordered a Raspberry Pi so I can start testing myself.

[AnotherDroog]

Great to see you investigating this area @NoahO!

Yes, CPU load was unexpectedly low during syncing. I like your hypothesis and we should test it!

My first instinct would be to suspect USB 2 maximum bandwidth of 60MB/s being the limiting factor and not the USB 3 drives we have being the bottleneck.

I haven’t benchmarked ours (Slim Fit model) but there’s Sandisk drives that comfortably write 60MB/s and read 100-200MB/s

_{Sent with GitHawk}

[noaho]

It's sequential write speeds of 60MB/s, not write speed in general.

To give an anecdotal example, I have a Sandisk USB3 USB Disk and a Samsung T3

When using the Sandisk over USB2, to play games, it is hugely slower than using the Samsung T3 (also over USB2)

The difference being one is designed for file transfers, and the other is a real SSD controller. The Sandisk write speeds are very fast when writing files sequentially to the card.

The youtube video demonstrates this better though.

[meeDamian]

It would indeed be amazing to investigate this!

[AnotherDroog]

You’re right, we can speed this up significantly, however, turning off synchronous writes would be very dangerous! At the very least I would expect us to have UPS.

[AnotherDroog]

Another data point,

Syncing with HDD and 3B+, CPU temp 56C, Load average: 4.31 4.55 4.60

[AnotherDroog]

When lnd is filtering blocks, e.g. from having to catch up a few days downtime, the CPU is pushed on multiple cores and it helps to ensure cpu frequency scaling is set correctly (e.g. ondemand) and no undervoltage occurs.

Active cooling reduced temps from >71C to <44C and noticeably improved performance

[AnotherDroog]

Using cheap SanDisk USB 2.0 USB drives:

tar is extracting at around 6MB/s. Since it's currently reading from the same device it's writing to (also 6MB/s), I wonder if we can improve the speed by using a second USB drive or the SD card instead.

[nolim1t]

No longer valid in this project. Perhaps this might be valid in noma?