search

filecoin-project / rust-fil-proofs

Proofs for Filecoin in Rust
Other
492 stars 314 forks source link

fix: make add_piece taking less time #1707

Closed vmx closed 1 year ago

vmx commented 1 year ago

add_piece operates in 64 bytes pieces. It's faster to operate in bigger chunks. This commits increases the buffer size to 4KiB. This makes adding a 32GiB piece about more than 2x faster. On the hardware I used it goes down from 14min to about 6min.

cryptonemo commented 1 year ago

Can we check if lotus is using this method? I recall it not being used, but may be wrong there. In any case, it should help speed up our big tests.

vmx commented 1 year ago

Can we check if lotus is using this method?

write_with_alignment seems to be the only API call in the FFI using it. It is then called WriteWithAlignment on the Go side. If I grep the Lotus repo for that, I only find it used in tests.

Why is that important? It's not a breaking change, it just improves things. So if Lotus would use it, it would also be beneficial for them.

cryptonemo commented 1 year ago

Can we check if lotus is using this method?

write_with_alignment seems to be the only API call in the FFI using it. It is then called WriteWithAlignment on the Go side. If I grep the Lotus repo for that, I only find it used in tests.

Why is that important? It's not a breaking change, it just improves things. So if Lotus would use it, it would also be beneficial for them.

I just wanted to know if we'd see the speed-ups outside of tests.