F3 Rust implementation tracking issue

ruseinov commented 4 months ago

Issue summary

This tracks our current progress on f3 integration.

Considerations for wrapping/re-write

Calling Go from Rust solutions: https://github.com/ihciah/rust2go and https://github.com/andeya/fcplug
Re-writing f3 in Rust.

Potential complications

Call Go code from Rust

We'll need a Go sidecar for that, we could then call it using rust2go, that should be easy enough.

Re-write

High level plan: https://hackmd.io/rXK7k6jOQQ6nxCo1RJMbfw
Repo: https://github.com/ChainSafe/rust-f3
This is going to need extensive testing, as we won't be able to rely on the test results of the reference implementation.
A significant time investment.

Roadmap

To minimize the timeline risk, we've decided to

[x] 1. implement F3 in forest with the go-f3 sidecar approach assuming go-f3 sidecar is almost ready
[x] 2. move go-f3 sidecar into Forest via FFI(rust2go) the number of go functions we need to make FFI for is small(less than 3). we've successfully made FFI for some go interop tests of p2p functionalities with rust2go
[ ] 3. (simultaneously) implement rust-f3 with proper testing
[ ] 4. switch from go-f3 to rust-f3

go-f3 integration with sidecar/FFI approach: #4644

rust-f3 implementation work items

[ ] Implement gp-bft primitives for other modules to use, as this package is depended upon by most others.
[ ] Implement blssig as it's unavoidable and pretty self-contained. the rest TBD

Fine-grained to-do for rust-f3

[x] #4693
[ ] #4705
[ ] #4828
[ ] #4829
[ ] #4830
[ ] #4831
[ ] #4832
[ ] #4833
[ ] #4834
[ ] #4835
[ ] #4836
[ ] #4873

Implementation diagram, helpful to see what can be parallelized.

flowchart TD
 subgraph s1["signing"]
        n3["blssig"]
        n5["gnark"]
  end
 subgraph s2["cert exchange"]
        n6["server"]
        n7["client"]
        n8["store"]
  end
 subgraph s3["Internal tools"]
        n12["power store"]
        n13["caching"]
        n14["measurements"]
        n11["wal"]
  end
    A["GPBFT"] --> I["certs"] & s1 & s2 & n9["EC Backend Interface"] & n12 & n16["f3 node"] & n15["manifest"]
    n5 --> n3
    s1 --> I
    I --> s2 & n16
    n8 --> n6
    s3 --> n16
    n9 --> n16
    s2 --> n16
    n15 --> n16 & n12
    n17["merkle"] --> A

     A:::done
    classDef done fill:#008000

Other information and links

Lotus f3 initial integration PR: https://github.com/filecoin-project/lotus/pull/12119
f3 info needs to end up in snapshots: https://github.com/filecoin-project/go-f3/issues/480
Snapshot versioning: https://docs.rs/forest-filecoin/latest/forest_filecoin/utils/db/car_stream/struct.CarHeader.html#structfield.version
Re-write plan: https://hackmd.io/rXK7k6jOQQ6nxCo1RJMbfw

LesnyRumcajs commented 3 months ago

@hanabi1224 @ruseinov Could you please update the issue to reflect the current plan?

ansermino commented 3 months ago

@hanabi1224 Thanks a bunch for updating this! Please can you breakout the go-f3 and the rust-f3 into their own issues/epics 🙏

Am I correct in understanding that integrating the sidecar is blocked until we get more info from the go-f3 team?

ruseinov commented 3 months ago

@hanabi1224 Thanks a bunch for updating this! Please can you breakout the go-f3 and the rust-f3 into their own issues/epics 🙏

Am I correct in understanding that integrating the sidecar is blocked until we get more info from the go-f3 team?

We have some new info in that regard. It seems like there's not going to be much action in terms of participation, because that depends on mining capabilities. Those in our case depend on Curio. So we're starting with following (getting all the certs and verifying them). What this means for Forest is:

Fast finality
F3 snapshot data, which basically is the whole cert sequence, which according to Kuba are lightweight.

It turns out that in order to follow we don't actually need much, just the cert exchange, which is pretty self-contained. So we'll be focusing on that and the sidecar in parallel. To be continued.

hanabi1224 commented 3 months ago

@hanabi1224 Thanks a bunch for updating this! Please can you breakout the go-f3 and the rust-f3 into their own issues/epics 🙏

Am I correct in understanding that integrating the sidecar is blocked until we get more info from the go-f3 team?

@ansermino Done. #4644

As @ruseinov mentioned, we're now clear about what are missing in the current go-f3 sidecar and I will look into closing the gap in a quick and dirty way to get a working sidecar demo

ansermino commented 3 weeks ago

@ruseinov What is the current state of this? Doesn't seem like anything has notably progressed in the last month+?

ruseinov commented 3 weeks ago

We've merged the second part of certs implementation a couple of weeks ago. The crypto package issue has been revisited to include fresh changes in go-f3, which have been introduced recently and involve a custom GNARK backend. I've been recently busy with other pressing tasks, but getting back to this exactly today to flesh out cert-exchange into more smaller issues and tackle those. By the looks of it we'll need to tackle certstore interface and MemoryDB implementation first to unblock certexchange implementation.

There's also a question of go-f3 being unstable, so I'm foreseeing some changes/bug-fixes in go-f3 in the near future which we will have to reflect in our codebase as well.

ChainSafe / forest