search

mirage / mirage-crypto

Cryptographic primitives for OCaml, in OCaml (also used in MirageOS)
ISC License
77 stars 43 forks source link

benchmarks: provide a digest benchmark which uses various implementations #202

Closed hannesm closed 6 months ago

hannesm commented 7 months ago

I wanted to easily compare different implementations, and started with hashing / digests.

In this PR, first the bench/speed.exe has a split-off helper.ml (no surprises) ;)

Then, I developed bench/digests.exe which tests nocrypto (https://github.com/hannesm/ocaml-nocrypto/tree/compiles - 0.5.4-2 plus allowing newer cstruct releases to avoid lots of changes in my switch), mirage-crypto (commit 6fa26bd2e501e2c7d4b61e7afd5297d4197a622d), digestif 1.1.4, ocaml-sha 1.15.4, cryptokit 1.19, OCaml 4.14.1 and openssl 3.0.12

All tests are run on my i7-5600U CPU @ 2.60GHz using FreeBSD-14, OCaml 4.14.1 (happy to see numbers on other architectures / processors):

Resuls:

MD5

size nocrypto mirage-cry digestif stdlib cryptokit openssl
16 51.682 63.650 78.144 107.970 90.273 59.678
64 148.387 172.181 195.855 230.107 205.056 169.489
256 313.978 332.245 357.565 385.844 361.911 364.497
1024 433.760 445.244 455.512 464.499 454.579 513.199
8192 486.566 489.529 491.404 492.987 490.263 587.799

First column in bytes, all others in MB/s (1MB = 1024 * 1024)

SHA1

size nocrypto mirage-cry digestif ocaml-sha cryptokit openssl
16 43.659 54.273 65.635 74.074 52.385 56.167
64 122.322 142.711 161.865 176.676 116.828 164.608
256 260.846 279.374 298.677 315.168 199.342 398.186
1024 364.759 375.541 385.924 393.510 245.253 614.528
8192 416.407 415.313 420.857 430.265 261.766 736.483

First column in bytes, all others in MB/s (1MB = 1024 * 1024)

SHA256

size nocrypto mirage-cry digestif ocaml-sha cryptokit openssl
16 26.480 31.774 34.243 36.839 34.678 38.340
64 69.416 77.409 82.467 81.558 76.503 99.244
256 131.065 139.294 142.987 137.850 129.533 219.244
1024 171.909 173.974 175.883 163.997 157.563 305.108
8192 188.027 188.982 188.704 180.706 168.621 341.120

First column in bytes, all others in MB/s (1MB = 1024 * 1024)

SHA512

size nocrypto mirage-cry digestif ocaml-sha cryptokit openssl
16 23.323 25.765 25.326 23.991 25.605 30.038
64 92.719 103.203 99.902 104.559 110.152 120.589
256 165.876 167.433 158.540 156.484 167.372 244.509
1024 243.956 237.645 235.108 230.279 234.339 387.885
8192 282.661 269.457 270.386 253.435 266.440 487.495

First column in bytes, all others in MB/s (1MB = 1024 * 1024)

hannesm commented 7 months ago

A table about X25519:

X25519

mirage-cryp callipyge rfc7748 hacl_x25519 openssl
1 15598.650 294.174 795.586 15579.182 23468.535

Note, the hacl_star package doesn't build on FreeBSD, thus it is not present above. And I patched the hacl_x25519 to use Cstruct.length (and thus a recent cstruct).

dinosaure commented 7 months ago

On AMD Ryzen 9 7950X (4.5 GHz):

SHA1

size nocrypto mirage-cryp digestif ocaml-sha cryptokit openssl
16 79.460 113.065 160.568 188.077 61.715 213.591
64 239.724 330.320 412.439 438.737 131.987 665.491
256 590.134 704.546 792.350 789.971 224.407 1545.151
1024 932.007 994.807 1024.803 973.503 264.234 2322.035
8192 1100.019 1110.046 1136.005 1040.819 282.564 2680.815

SHA256

size nocrypto mirage-cryp digestif ocaml-sha cryptokit openssl
16 56.059 63.366 85.261 95.510 86.249 203.169
64 150.514 176.322 191.899 211.134 188.266 632.563
256 304.330 328.814 340.239 364.059 316.101 1460.627
1024 412.551 421.731 419.593 445.085 382.492 2173.267
8192 456.972 456.577 451.432 476.765 413.332 2551.422

SHA512

size nocrypto mirage-cryp digestif ocaml-sha cryptokit openssl
16 48.668 54.247 70.859 75.767 68.872 91.953
64 191.719 213.347 280.200 299.890 295.584 368.394
256 373.461 395.111 440.088 460.356 445.883 678.256
1024 587.753 606.742 620.436 657.357 614.305 1001.971
8192 705.427 714.307 707.548 750.250 692.852 1171.290
palainp commented 7 months ago

According to /dev/cpuinfo my vcpus are i7-1365U running at 2688.006Hz, and the laptop was wallpluged when benchmarking. Here are my results but it's hard to understand the different behavior, MD5 results, as well as nocrypto, mirage-crypto, digestif and cruptokit, are quite similar to @hannesm, but openssl significantly outperforms on the SHA1 and 256 suites:

MD5

size nocrypto mirage-cryp digestif stdlib cryptokit openssl
16 46.065 60.942 77.967 87.102 78.350 70.518
64 132.995 159.852 191.998 198.498 186.323 169.594
256 278.367 309.204 339.583 345.299 334.919 330.738
1024 396.499 413.291 420.163 424.478 420.437 447.872
8192 441.823 450.029 451.587 454.835 454.077 499.858

SHA1

size nocrypto mirage-cryp digestif ocaml-sha cryptokit openssl
16 52.550 54.621 69.878 69.416 33.155 78.529
64 117.249 141.327 165.596 155.706 70.543 285.192
256 249.779 275.922 300.234 265.001 117.018 793.100
1024 353.590 370.176 376.592 321.868 140.868 1426.052
8192 399.865 404.476 405.558 343.124 149.753 1859.459

SHA256

size nocrypto mirage-cryp digestif ocaml-sha cryptokit openssl
16 27.268 32.401 36.046 39.772 36.702 76.896
64 68.729 76.204 80.636 88.043 78.884 271.642
256 130.334 136.683 139.198 151.284 131.043 721.106
1024 168.792 172.998 171.136 183.977 158.319 1229.280
8192 186.274 187.169 183.687 196.986 168.498 1547.014

SHA512

size nocrypto mirage-cryp digestif ocaml-sha cryptokit openssl
16 21.449 23.902 26.861 30.971 30.004 26.553
64 84.842 96.364 107.193 124.293 120.362 108.819
256 160.448 168.171 174.807 189.678 182.127 217.158
1024 240.432 250.623 248.739 266.476 250.449 347.056
8192 288.568 289.504 287.682 302.394 282.529 419.953
hannesm commented 7 months ago

hmm, was there some trick to add some private opam package with the dependencies to get this compiling in CI -- while "private" means that a dune-publish does not accidentally publish that opam package? or is that just something I dreamed of - and we can either live with "ocaml-ci" not finding the dependencies and failing, or add another opam package where we need to be careful to never publish it in a release?

hannesm commented 6 months ago

since digest algorithms have been removed from mirage-crypto, this is no longer very relevant. developing benchmarks for the various ocaml libraries and openssl as baseline is still on our list, and will hopefully land soon in this repository :)