search

jimsalterjrs / sanoid

These are policy-driven snapshot management and replication tools which use OpenZFS for underlying next-gen storage. (Btrfs support plans are shelved unless and until btrfs becomes reliable.)
http://www.openoid.net/products/
GNU General Public License v3.0
3.14k stars 308 forks source link

mbuffer: warning: high value of number of blocks(32768): increase block size for better performance #947

Open tomerh2001 opened 3 months ago

tomerh2001 commented 3 months ago

I am trying to perform a complete restore of my ZFS pool to my fresh server via this command:

syncoid                                 \
    -r                                  \
    --no-privilege-elevation            \
    --mbuffer-size=4G                   \
    --no-sync-snap                      \
    root@zh4158b.rsync.net:data1        \
    Pool/restore

I have free 58GB of RAM, so I could use that to speed up the process.

This is the output I'm getting from running this command:

homeserver@homeserver:/app/scripts$ sudo sh restore.sh
[sudo] password for homeserver: 
NEWEST SNAPSHOT: syncoid_homeserver_2024-08-20:22:36:48-GMT03:00
INFO: no snapshots on source newer than syncoid_homeserver_2024-08-20:22:36:48-GMT03:00 on target. Nothing to do, not syncing.
NEWEST SNAPSHOT: syncoid_homeserver_2024-08-20:22:43:44-GMT03:00
Sending incremental data1/Data@autosnap_2024-04-08_06:00:00_hourly ... syncoid_homeserver_2024-08-20:22:43:44-GMT03:00 (~ 3.2 MB):
mbuffer: warning: high value of number of blocks(32768): increase block size for better performance
mbuffer: warning: high value of number of blocks(32768): increase block size for better performance
mbuffer: warning: allocating more than half of available memory                                                  
264KiB 0:01:36 [8.22KiB/s] [=====>                                                                         ]  8% ETA 0:22:54

I get this warning: mbuffer: warning: high value of number of blocks(32768): increase block size for better performance, together with extremely slow write speeds...

So:

  1. How do I increase the block size it's complaining about?
  2. How do I increase the write speed... I can't restore 2TB like that...

These are some benchmarks I ran to show my hardware is perfectly capable of faster restore:

homeserver@homeserver:/app/scripts$ sudo sh benchmark.sh 
---------------------
1. Internet Speed Test
---------------------
Download Speed: 1940.50 Mbit/s
Upload Speed: 235.01 Mbit/s

---------------------
2. CPU Performance Test (FLOPS)
---------------------
CPU Performance: 6158.83 FLOPS

---------------------
3. RAM Speed Test
---------------------
RAM Write Speed: 10240.00 MiB/s
RAM Read Speed: 10240.00 MiB/s

---------------------
4. Disk Write and Read Speed Test
---------------------
Disk Write Speed: 1.8 GB/s
Disk Read Speed: 3.5 GB/s

---------------------
5. ZFS Pool Write and Read Speed Test
---------------------
ZFS Pool Write Speed: 2.8 GB/s
ZFS Pool Read Speed: 3.8 GB/s
jimsalterjrs commented 3 months ago

You'd have to go into the source code to change mbuffer's arguments to use larger blocks. Doing so will potentially improve total throughput, while potentially making resumes more painful if you lose connection in the middle of a sync.

More importantly, though...

mbuffer: warning: allocating more than half of available memory

If mbuffer is allocating more than half of what memory you've got left out of 58GiB, something is badly wrong and you may need to take a closer look at what you're doing with the RAM on that system.