Nugine / s3s

S3 Service Adapter
Apache License 2.0
144 stars 34 forks source link

Added Openssl feature #125

Closed nomick closed 9 months ago

nomick commented 9 months ago

Changing the sha2 implementation to openssl may increase the speed tremendously. On a Cascade Lake Xeon we saw 60% increase in PUT performance. As openssl may not be a good choice in every situation, it is capsuled in a cargo feature. The unwraps may be not wanted, but avoiding them would change the function signature.

WARP before:

Operation: PUT. Concurrency: 20

Throughput, split into 111 x 1s:

WARP after:

Operation: PUT. Concurrency: 20

Throughput, split into 65 x 1s:


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Nugine commented 9 months ago

So the sha256 in openssl is much faster than pure rust sha2? It's suprising. We should add a simple benchmark to track this difference and report to RustCrypto/hashes/sha2.

nomick commented 9 months ago

Unfortunately. they are perfectly aware of this. See https://github.com/RustCrypto/hashes/issues/327

St4NNi commented 9 months ago

Was curious about this, I guess it really depends on your specific CPU. If your CPU supports the SHA-NI extension openssl and RustCrypto perform about the same, if not RustCrypto is up to 2x slower.

A quick test on our EPYC Milan (Zen3) and on my personal Zen4 machine showed, as expected, no measurable difference between RustCrypto and OpenSSL.