search

restic / rest-server

Rest Server is a high performance HTTP server that implements restic's REST backend API.
BSD 2-Clause "Simplified" License
943 stars 140 forks source link

Server-side checksumming #299

Open santiagobiali opened 3 weeks ago

santiagobiali commented 3 weeks ago

Output of rest-server --version:

rest-server version rest-server 0.13.0 compiled with go1.22.5 on linux/amd64

What should rest-server do differently?

Add a function in rest-server that calculates the sha256sum of a given file in the repository and returns the checksum.

What are you trying to do? What is your use case?

Scenario: a server running rest-server <-- limited network bandwidth --> a client running Restic client. I'm trying to add more consistency to the restic check without needing to download the whole repository to the client's machine. As every file inside the folder /repository/data has it's sha256sum as it's own name, and the checksum is known for every file. Restic could use it to detect corrupted files when running restic check --checksum-only. This way, a basic check could be performed on the whole repository without the need to transfer all the data from the server to the client to check.

This check isn't as complete as the default one because it doesn't decrypt the data, but it's way faster.

And yes, some logic will also be needed to be implemented on the Restic client, but as it's this check is only possible if using the rest-server, I found this github repository to be the most reasonable to discuss this feature.

Did rest-server help you today? Did it make you happy in any way?

wojas commented 3 weeks ago

This would be a useful addition.

Supporting this on the rest-server needs to be optional, as it's a heavy operation on some deployments, in which case restic should fall back to the old way. See also the discussion in https://github.com/restic/rest-server/pull/130#issuecomment-753966573

Unlike a client-side check, this requires you to trust the server, so perhaps this needs to be opt-in on the client side. The downside of that is that most users will not discover this extra flag. Perhaps recommend it in the log output when not used?

santiagobiali commented 2 weeks ago

Unlike a client-side check, this requires you to trust the server, so perhaps this needs to be opt-in on the client side.

Yes, optional flag on the client side seems to be the way to go. It should be ok to trust the server, perhaps add a few extra steps to increase the trust and prevent some false positives:

Calculating the checksums doesn't even need to have the lock on the repository. BUT if some checksum comparison fails, then the lock should be requested and calculated again. This would prevent files still being written to have the "wrong" checksum.

CPU performance should not be an issue, even my router can calculate hashes faster than sata SSDs can read. As we can see here:

openssl speed -multi 1 sha256; changing multi from 1 to xx: The 'numbers' are in 1000s of bytes per second processed. device type multi 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
rpi-4b sha256 1 13767.78k 40242.32k 87198.12k 123272.53k 140798.63k 142191.27k
rpi-4b sha256 2 27756.68k 79921.37k 174163.03k 246635.18k 281864.87k 284519.08k
rpi-4b sha256 3 39660.10k 115378.43k 254801.07k 366981.80k 421827.93k 424618.67k
rpi-4b sha256 4 55653.35k 159964.97k 348450.22k 493146.79k 562462.72k 567694.68k
AX6S sha256 1 10985.35k 40858.56k 135536.55k 322297.17k 539677.45k 563118.08k
AX6S sha256 2 21214.01k 80436.07k 264237.40k 634267.31k 1063723.01k 1116099.93k
Xeon sha256 1 97163.26k 214521.07k 404580.35k 508192.77k 548588.20k 543069.53k
Xeon sha256 2 194668.87k 435993.43k 812625.75k 1014624.60k 1100769.96k 1107050.50k
Xeon sha256 4 392128.25k 865436.80k 1599896.06k 2024864.09k 2190772.91k 2200436.74k
Xeon sha256 8 779188.17k 1727125.65k 3220268.54k 3988109.65k 4269596.67k 4318079.66k
Xeon sha256 16 1456016.65k 2898188.01k 6044620.49k 7398187.35k 8230584.32k 7811629.06k
Xeon sha256 32 1487867.32k 3288986.79k 6348034.21k 8115302.06k 8995089.07k 8937854.29k
i5 sha256 1 116204.62k 358362.75k 853148.16k 1322539.01k 1584401.07k 1603780.61k
i5 sha256 2 224139.27k 706349.63k 1715574.53k 2677368.83k 3226146.13k 3253381.80k
i5 sha256 4 444497.54k 1394851.01k 3400928.26k 5270369.96k 6343464.28k 6442603.86k
i5 sha256 8 572761.01k 1711124.33k 5021131.61k 7901587.80k 10305727.15k 10504432.30k
i5 sha256 12 618255.57k 1942213.40k 5362399.40k 9510028.92k 11949817.08k 12330631.17k
i7 sha256 1 49676.98k 141419.37k 308914.35k 436739.41k 493264.90k 487877.29k
i7 sha256 2 110710.93k 271003.61k 578058.50k 820746.24k 927602.01k 885025.45k
i7 sha256 4 154385.42k 371080.23k 1085178.62k 1245001.73k 1477017.60k 1675449.69k
i7 sha256 8 182294.67k 514043.65k 1176136.36k 1613019.79k 1817834.84k 1890407.77k

The 40MB/s speed measured on PR/130 is actually the SD-Card speed, and not how much hashing the rpi4b is capable of.

Specs:

Raspberry-pi 4B (4 cores)
    OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
    options: bn(64,64)
    compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-928mA1/openssl-3.0.13=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
    CPUINFO: OPENSSL_armcap=0x81

Xiaomi AX6S (Router), CPU is a MediaTek MT7622B (2 cores, 1,35GHz)
    OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
    options: bn(64,64)
    compiler: aarch64-openwrt-linux-musl-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -pipe -mcpu=cortex-a53+crypto+crc -funsafe-math-optimizations -fno-plt -fhonour-copts -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -O3 -DPIC -fPIC -pipe -mcpu=cortex-a53+crypto+crc -funsafe-math-optimizations -fno-plt -fhonour-copts -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -O3 -fPIC -fuse-ld=bfd -znow -zrelro -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED -DNDEBUG -D_FORTIFY_SOURCE=1 -DPIC
    CPUINFO: OPENSSL_armcap=0x3d

2x Xeon Gold 6244 (2x 8 cores, 16 threads => 16 cores, 32 threads)
    OpenSSL 1.1.1w  11 Sep 2023
    options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr)
    compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-1.1.1w=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2

Intel i5-11400 (6 cores, 12 threads)
    OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
    options: bn(64,64)
    compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.13=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
    CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x405f5ef2bf67eb

Intel i7-4790K (4 cores, 8 threads)
    OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
    options: bn(64,64)
    compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.13=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
    CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x27ab
wojas commented 1 week ago

CPU performance should not be an issue, even my router can calculate hashes faster than sata SSDs can read. As we can see here:

This is true for almost all modern devices that have SHA256 acceleration, but not necessarily for older devices. If I remember correctly, my 2013 ARM7 Synology NAS took 20s just to decrypt the restic private key to run a local restic backup, and could only hash at something like 3 MB/s.

There should be at least a way to opt-out in rest-server, and maybe require an opt-in.