meta: backup

[jiefenghuang]

Goal

A new binary backup method enables more efficient and memory-saving dump and load operations, producing smaller backups. It is designed for large-scale file system backups and migrations.

Limited

• Dump: For DB (single concurrency) and TiKV, data consistency is ensured through snapshot reads. In other scenarios, such as DB (with multiple concurrency) and Redis, backups need to be exported in read-only mode.

Performance

data set: 1m files goos: linux goarch: amd64 cpu: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics

• mysql
  • dumpv1(non-fast): 49.87s
  • dumpv1(fast): 18.61s
  • dumpv2(CoNum=1): 16.93s
  • dumpv2(CoNum=10): 4.18s
  • loadv1: 72.06s
  • loadv2(CoNum=10): 32.70s
• redis
  • dumpv1: 4.52s, 3.96s ...
  • dumpv2(CoNum=1): 3.65s
  • dumpv2(CoNum=10): 3.45s
  • loadv1: 7.30s
  • loadv2(CoNum=10): 1.32s
• tikv (tiup playground)
  • dumpv1(non-fast): 54.60s
  • dumpv1(fast): 7.16s
  • dumpv2(CoNum=1): 1.81s
  • dumpv2(CoNum=more than one): TODO: need a TiKV cluster with multiple disks to check this
  • loadv1: 9.64s
  • loadv2(CoNum=10): 5.42s

    
    BenchmarkLoadDumpV2/Load_mysql-16                      1        72060549327 ns/op       9986896728 B/op 137978053 allocs/op
    BenchmarkLoadDumpV2/Dump_non-fast_mode_mysql-16                        1        49878605404 ns/op       19085003992 B/op     554803266 allocs/op
    BenchmarkLoadDumpV2/Dump_fast_mode_mysql-16                            1        18618667068 ns/op       7130212416 B/op      292465453 allocs/op
    BenchmarkLoadDumpV2/DumpV2_CoNum=1_mysql-16                            1        16930252356 ns/op       7911537616 B/op      279367159 allocs/op
    BenchmarkLoadDumpV2/DumpV2_CoNum=10_mysql-16                           1        4186731500 ns/op        7781710912 B/op      278340145 allocs/op
    BenchmarkLoadDumpV2/LoadV2_mysql-16                                    1        32705284621 ns/op       8056306592 B/op 169484502 allocs/op

BenchmarkLoadDumpV2/Load_redis-16 1 7307923118 ns/op 2062659712 B/op 27617610 allocs/op BenchmarkLoadDumpV2/Dump_non-fast_mode_redis-16 1 4528533637 ns/op 2303354744 B/op 41546163 allocs/op BenchmarkLoadDumpV2/Dump_fast_mode_redis-16 1 3967532937 ns/op 2299847272 B/op 41513331 allocs/op BenchmarkLoadDumpV2/DumpV2_CoNum=1_redis-16 1 3659069323 ns/op 712397208 B/op 9024910 allocs/op BenchmarkLoadDumpV2/DumpV2_CoNum=10_redis-16 1 3456428554 ns/op 713347296 B/op 9033111 allocs/op BenchmarkLoadDumpV2/LoadV2_redis-16 1 1323064794 ns/op 1063459224 B/op 15249467 allocs/op

BenchmarkLoadDumpV2/Load_tikv-16 1 9644265753 ns/op 2708921496 B/op 28191080 allocs/op BenchmarkLoadDumpV2/Dump_non-fast_mode_tikv-16 1 54607733112 ns/op 14235384304 B/op 256777839 allocs/op BenchmarkLoadDumpV2/Dump_fast_mode_tikv-16 1 7169758572 ns/op 2514788216 B/op 34111266 allocs/op BenchmarkLoadDumpV2/DumpV2_CoNum=1_tikv-16 1 1812321521 ns/op 846469264 B/op 8966257 allocs/op BenchmarkLoadDumpV2/LoadV2_tikv-16 1 5420456001 ns/op 991102312 B/op 10047605 allocs/op

## Memory

- mysql (TODO: optimise)
<img width="1096" alt="image" src="https://github.com/user-attachments/assets/a19b8723-3cb1-4504-a822-0cc276d66d74">

- redis
<img width="1028" alt="image" src="https://github.com/user-attachments/assets/bdcf2e3c-818a-40ab-bae9-65d3977efbe6">

- tikv
<img width="771" alt="image" src="https://github.com/user-attachments/assets/c8846059-5068-4a8b-ada2-192b5bc62f1f">