celestiaorg / go-square

A library for encoding blobs into a 2D square of evenly sized chunks designed for sampling and reconstruction
Apache License 2.0
13 stars 16 forks source link

perf(merkle): optimize hashing by reusing sha256 in a pool #44

Closed odeke-em closed 2 months ago

odeke-em commented 6 months ago

This change optimizes SHA256 hashing for leafHash & innerHash by pooling and reusing hashes, which massively reduces on RAM consumption and even CPU time (benchmarking on my noisy machine is skewed but the results can be verified individually) producing results below:

$ benchstat before.txt after.txt
name                          old time/op    new time/op    delta
HashAlternatives/recursive-8    77.6µs ± 2%    77.0µs ± 2%     ~     (p=0.165 n=10+10)
HashAlternatives/iterative-8    76.3µs ± 1%    76.2µs ± 3%     ~     (p=0.720 n=9+10)

name                          old alloc/op   new alloc/op   delta
HashAlternatives/recursive-8    25.4kB ± 0%     6.4kB ± 0%  -74.94%  (p=0.000 n=10+10)
HashAlternatives/iterative-8    28.1kB ± 0%     9.1kB ± 0%  -67.78%  (p=0.000 n=10+10)

name                          old allocs/op  new allocs/op  delta
HashAlternatives/recursive-8       497 ± 0%       199 ± 0%  -59.96%  (p=0.000 n=10+10)
HashAlternatives/iterative-8       498 ± 0%       200 ± 0%  -59.84%  (p=0.000 n=10+10)

Fixes #43

Overview

Checklist

odeke-em commented 6 months ago

@rootulp how come the code was copied and pasted in here without the appropriate attributions/licenses or without just plain imports or submodules?

rootulp commented 6 months ago

I think this is relevant:

https://github.com/celestiaorg/go-square/blob/4e84d80f3e55ac3093005c8089186d68ee62541d/merkle/README.md?plain=1#L6

I suppose we could also copy + paste this license into that directory.

cmwaters commented 2 months ago

Closing this because for now we've decided to keep the merkle library in our cometbft fork. Thanks for opening the PR in any case. Sorry that we didn't merge it. We may look to apply this to our cometbft fork in the future