F3 Rust implementation tracking issue

[ruseinov]

Issue summary

This tracks our current progress on f3 integration.

Considerations for wrapping/re-write

• Calling Go from Rust solutions: https://github.com/ihciah/rust2go and https://github.com/andeya/fcplug
• Re-writing f3 in Rust.

Potential complications

Call Go code from Rust

• We'll need a Go sidecar for that, we could then call it using rust2go, that should be easy enough.

Re-write

• High level plan: https://hackmd.io/rXK7k6jOQQ6nxCo1RJMbfw
• Repo: https://github.com/ChainSafe/rust-f3
• This is going to need extensive testing, as we won't be able to rely on the test results of the reference implementation.
• A significant time investment.

Roadmap

To minimize the timeline risk, we've decided to

• [ ] 1. implement F3 in forest with the go-f3 sidecar approach assuming go-f3 sidecar is almost ready
• [ ] 2. move go-f3 sidecar into Forest via FFI(rust2go) the number of go functions we need to make FFI for is small(less than 3). we've successfully made FFI for some go interop tests of p2p functionalities with rust2go
• [ ] 3. (simultaneously) implement rust-f3 with proper testing
• [ ] 4. switch from go-f3 to rust-f3

go-f3 integration with sidecar/FFI approach: #4644

rust-f3 implementation work items

• [ ] Implement gp-bft primitives for other modules to use, as this package is depended upon by most others.
• [ ] Implement blssig as it's unavoidable and pretty self-contained. the rest TBD

Fine-grained to-do for rust-f3

• [ ] #4693
• [ ] #4705
• [ ] Implement GPBFT primitives that certexchange package uses and port certexchange

Other information and links

• Lotus f3 initial integration PR: https://github.com/filecoin-project/lotus/pull/12119
• f3 info needs to end up in snapshots: https://github.com/filecoin-project/go-f3/issues/480
• Snapshot versioning: https://docs.rs/forest-filecoin/latest/forest_filecoin/utils/db/car_stream/struct.CarHeader.html#structfield.version
• Re-write plan: https://hackmd.io/rXK7k6jOQQ6nxCo1RJMbfw

[LesnyRumcajs]

@hanabi1224 @ruseinov Could you please update the issue to reflect the current plan?

[ansermino]

@hanabi1224 Thanks a bunch for updating this! Please can you breakout the go-f3 and the rust-f3 into their own issues/epics 🙏

Am I correct in understanding that integrating the sidecar is blocked until we get more info from the go-f3 team?

[ruseinov]

@hanabi1224 Thanks a bunch for updating this! Please can you breakout the go-f3 and the rust-f3 into their own issues/epics 🙏

Am I correct in understanding that integrating the sidecar is blocked until we get more info from the go-f3 team?

We have some new info in that regard. It seems like there's not going to be much action in terms of participation, because that depends on mining capabilities. Those in our case depend on Curio. So we're starting with following (getting all the certs and verifying them). What this means for Forest is:

1. Fast finality
2. F3 snapshot data, which basically is the whole cert sequence, which according to Kuba are lightweight.

It turns out that in order to follow we don't actually need much, just the cert exchange, which is pretty self-contained. So we'll be focusing on that and the sidecar in parallel. To be continued.

[hanabi1224]

@hanabi1224 Thanks a bunch for updating this! Please can you breakout the go-f3 and the rust-f3 into their own issues/epics 🙏

Am I correct in understanding that integrating the sidecar is blocked until we get more info from the go-f3 team?

@ansermino Done. #4644

As @ruseinov mentioned, we're now clear about what are missing in the current go-f3 sidecar and I will look into closing the gap in a quick and dirty way to get a working sidecar demo