New Feature: Benchmark USB stick read IOPS and warn if sync will go slowly.

[BenWestgate]

do we have the ability to check if the space available is on SSD rather than HDD? my mean, it will be useful to at least hint to users that datadir location on HDD will be the cause of several weeks IBD, so, the user can decide to move to another machine or wait

Originally posted by @pythcoiner in https://github.com/wizardsardine/liana/issues/570#issuecomment-1635642239

Bails could do a random read test of the storage device at startup and warn users when their particular storage size, memory size and storage speed combo will result in sync greater than a week.

[github-actions[bot]]

Stale issue message

[BenWestgate]

Reducing priority to low because AssumeUTXO will make sync speed less critical.

Will need a good bash or python tool to do the Read IOPS test that does NOT require admin rights. My attempts have been inaccurate.

[github-actions[bot]]

Stale issue message

[BenWestgate]

Two factors: write speed which we can benchmark during the creation of the file filled with random data.

This has a minimum to be able to write the blockchain to disk fast enough to do 600GB in a week.

Meanwhile IOPS needs to be several thousand or 10MB/s of 4k qd=1 random reads to have a chance of a quick sync when the entire chainstate cannot fit in ram. Currently requires 16GB to make the random reads irrelevant.

[BenWestgate]

Give this code a test and see if it is remotely accurate:

Yes, it is possible to benchmark random read IOPS in Python without needing administrative rights. You can achieve this by creating a large file and performing random reads within that file. Here’s a simple example to get you started:

import os
import random
import time

# Configuration
file_path = "test_file.bin"
file_size = 1024 * 1024 * 1024 # 1 GB
block_size = 4096 # 4 KB
num_reads = 100000

# Create a large file if it does not exist
if not os.path.exists(file_path):
with open(file_path, "wb") as f:
f.write(os.urandom(file_size))

# Benchmark random read IOPS
with open(file_path, "rb") as f:
start_time = time.time()
for _ in range(num_reads):
offset = random.randint(0, file_size - block_size)
f.seek(offset)
f.read(block_size)
end_time = time.time()

# Calculate IOPS
elapsed_time = end_time - start_time
iops = num_reads / elapsed_time
print(f"Random read IOPS: {iops}")

This script creates a 1 GB file if it doesn't already exist and then performs 100,000 random reads of 4 KB blocks within that file, measuring the time taken to calculate the IOPS.

You can adjust file_size, block_size, and num_reads based on your specific requirements. This method benchmarks the IOPS at the file system level, which is sufficient for many applications.

On Tue, May 28, 2024 at 03:33, github-actions[bot] @.***(mailto:On Tue, May 28, 2024 at 03:33, github-actions[bot] < wrote:

Stale issue message

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>