search

filecoin-project / rust-fil-proofs

Proofs for Filecoin in Rust
Other
490 stars 314 forks source link

Avoid last `data` duplication in `transform_and_replicate_layers` #504

Closed schomatis closed 4 years ago

schomatis commented 5 years ago

After an initial study of the transform_and_replicate_layers function for https://github.com/filecoin-project/rust-fil-proofs/issues/481, one optimization that although minor seems worth doing would be to avoid duplicating the data that will be send to the outer_rx (as data_copy) in the last iteration,

https://github.com/filecoin-project/rust-fil-proofs/blob/90bb0445709a7ac38fefbeb100ed58e86d28f84b/storage-proofs/src/layered_drgporep.rs#L376-L377

since there won't be a vde::encode to consume it,

https://github.com/filecoin-project/rust-fil-proofs/blob/90bb0445709a7ac38fefbeb100ed58e86d28f84b/storage-proofs/src/layered_drgporep.rs#L397-L403 so we could just send data instead of data_copy, that is, we would somewhat unwind the last iteration. However, it's not clear to me how to do that, since Rust has no way to know that the last iteration won't actually use data (and hence it won't let me just take ownership of it to send it through the spawned thread).

@dignifiedquire WDYT? Is this a valid memory optimization? If so, how would you suggest restructuring the loop?

dignifiedquire commented 4 years ago

not an issue anymore