🚧 | B4: Integrate IPC and Mir into a production system

@jsoares commented on Wed Nov 24 2021

Description

Once we have a production-level implementation of hierarchical consensus (#6) and efficient subnet consensus protocols (#9), the next step is to combine them into a production system, deployable on the Filecoin mainnet.

Scope

Interplanetary Consensus (IPC) is a framework to enable on-demand horizontal scalability of Filecoin through the deployment of subnets (self-governing chains) that spawn their own state, validate messages in parallel, and seamlessly interact with any network in the hierarchy, as well as the Filecoin mainnet. Subnets can run different consensus algorithms, adjusting to the needs of the application. IPC is complementary to the programmability introduced by FVM, in that it provides a framework to further program the Filecoin network, accommodating a wide variety of use cases while overcoming potential consensus bottlenecks.

This project combines the outputs of #6 and #9 into a production implementation of IPC. It ends with actual real-world deployment but, as a first step towards that, it will launch Spacenet.

Dependencies

Roadmap

eta: 2023-12-31 children:

Talks

2022-10-27, Interplanetary Consensus: Scaling Blockchains with Subnets in an Easy-to-Trust Model

2022-10-27, Interplanetary Consensus: Now Playing in Eudico Near You

2022-10-27, Applications of IPC

2022-10-27, Building Modular and Scalable BFT Consensus Using the Mir Framework

Demos

2022-10-14, uptimeChecker — an FVM actor & service for cluster management

2022-12-05, Spacenet

2022-12-05, IPC FVM Actors Implementation

2023-03-21, Running an IPC Subnet

Resources

2022-09-26

✋ Attendees

async

📣 Updates

New Mir high-level documentation
@sergefdrv
- Working on feature parity between LegacyVM- and FMV-based IPC actors
- Learning Rust

🧵 Discussion

🎯 Up next

@adlrocha:
- Consensus interface PR ready for review EoW.
- BuilderNet planning and populate issues in the repo.
- Kick-off Mir integration in Lotus.
@sergefdrv
- Finish with feature parity of Join and Leave in FVM-based IPC actors
@matejpavlovic to revisiting Mir tasks

2022-10-03

✋ Attendees

async

📣 Updates

@sergefdrv
- Adjusted subnet join parameter in Eudico: itests still fail (timeout)
  - itests take long time to re-run (~10m)
- Draft of Support for Mir integration in FVM-based subnet actor
@matejpavlovic
- Updated high-level Mir documentation. This was crucial to onboard any potential helpers.
- Mir instruction videos: An introductory one and one where I write actual code, implementing a dummy ping-pong protocol with Mir completely from scratch (starting from a blank VM).
  - The sample code is also in the Mir repo.
- Partial Mir input message sanitization dropping faulty messages addressed to non-existent modules.
- Implemented backpressure on Mir-internal events. (Thanks for review @sergefdrv.) Can be merged right after Sergey's final look.
@adlrocha
- BuilderNet planning
  - Issues and (yet another) lotus fork for Mir integration
- Consensus interface PR discussed internally, and PR opened upstream (experimental/fvm-m2).
- Kicked-off Mir/Lotus integration
@dnkolegov:
- Mir network transport PR
  - requested additional review from Alfonso and Sergey
  - need to fix all the comments
    🎯 Up next
@sergefdrv
- Figure out why itests fail with adjusted subnet join parameter
- Finalize Support for Mir integration in FVM-based subnet actor
@matejpavlovic
- State synchronization and block validation protocol proposal
- Performance numbers
@adlrocha:
- Get PR for Mir integration in Lotus ready
- Next steps and design for full-node and "orthogonal" storage.
@dnkolegov:
- fix all the comments for PR 230
- discuss with @matejpavlovic the next task but it probably will be the client watermarking mechanism

2022-10-10

📣 Updates

@cryptoAtwill
- Uptime checker:
  - Uptime-checker demo + video recording
  - Finalized uptime-checker in shipyard and added preliminary test cases: Pending PR
- FVM Util:
  - Started reading and understanding ref-fvm and its underlying mechanism.
  - Migrate tcid
@matejpavlovic
- Block syncing and verification protocol description draft
- Outputting checkpoints from Mir This involves separating the checkpoint creation protocol from ISS that it was integral part of before.
- Some technical Mir-related updates:
  - Node config validation
  - More flexible SMR system setup
  - Temporary own address inclusion checking (using libp2p, the multiaddress is augmented and seems to not be included in the membership)
- Getting Started section in Mir doc for onboarding
- A lot of debugging of Mir/Lotus integration with @adlrocha and @dnkolegov
@dnkolegov:
- resolving comments and improving libp2p transport in Mir
- debugging of Mir-Lotus integration
- code review
@adlrocha
- Mir is miningin Lotus for a single node! Thank you Denis and Matej for your support. With more than one node it still gets stuck, let's sync after the weekly if you don't mind (as mentioned in Slack, I'm mainly concerned about tracking errors involving Mir from Lotus).
- BuilderNet spec
- Migrated all of our forks of FVM, the IPC actors, etc. from my personal repo to consensus-shipyard.
@sergefdrv
- Fixed issues with Adjusting subnet join parameter in Eudico
- Validator struct cleanup in Eudico
- Support for Mir integration in FVM-based subnet actor
- Code review of new Mir network transport module impl

🧵 Discussion

🎯 Up next

@cryptoAtwill
- Test MockedRuntime with fvm-* crate integration solution.
@matejpavlovic
- Preliminary WAN deployment and performance (finally...)
- Improve error handling / output in Mir to enable proper debugging with Lotus.
@dnkolegov:
- finish and merge libp2p PR
@adlrocha:
- Finalize Lotus/Mir integration for mining.
- Implement syncing protocol for Mir in Lotus.
- Bonus: Decouple storage from consensus.
@sergefdrv
- Dive into Atomic primitives and sample actor with support for atomic execution.

2022-10-17

📣 Updates

@cryptoAtwill
- FVM Util:
  - Build/modified existing runtime crate to ensure example code runs in wasm testnet also in unit tests. The previous runtime crate cannot build to wasm target due to missing c lib. Need to downgrade getrandom and rand temporarily to make it work.
  - Setup fvm local testnet env and ensured example crate compiles and runs.
  - Introduced cicd for checking cargo fmt/build/test/clippy
@adlrocha
- Integration of Mir in Lotus, decoupling of storage from consensus, and support for multiple validators. PR to lotus masterready.
- Kicked-off implementation of syncing protocol and full nodes compatible with Mir consensus.
- Rebased all of our active PRs (1, 2) in lotus and our fork at consensus-shipyard to target master (experimental/fvm-m2 will soon be deprecated).
- New fvm-utils repo with abstraction tools to ease the development of user-defined actors. Key stepping stone to make the IPC gateway user-defined
@matejpavlovic
- Some technical Mir-related updates:
- Preliminary WAN deployment code and benchmarks
  - TL;DR: Looks rather good. 2.5kreq/s max throughput on 8 nodes world-wide, unsaturated (100req/s) latency < 1s.
- New Mir networking module design
@sergefdrv
- Digging into Atomic primitives and sample actor with support for atomic execution
- Identified a potential issue with the current atomic execution scheme
- Proposed a new scheme to solve the issue
@dnkolegov
- Merged network improvements - PR (merged)
- Worked on the error handling and logging in Mir - PR (merged)
- Worked on the integration of Mir in Lotus (code review, resolving comments, refactoring) (rebased)
- Worked on syncing Lotus and Eudico code - PR

🧵 Discussion

@adlrocha: @matejpavlovic, @dnkolegov, according to our Spacenet plan, the target was to have all the implementation ready pre-LabWeek, and start testing the code post-LabWeek. What are we missing from the Mir side?
- @matejpavlovic: I went through the Issues and the only critical functionality-related item left is restarting from checkpoints. (The unified mempool and request pool can be closed, right, @dnkolegov?) The client watermarks are important too, but I guess the system can, for the very start, work without. It's just that a client will only be allowed to have one in-flight TX. One could consider this to be a performance thing, rather than functionality. All the other issues are related to robustness / testing and performance.

🎯 Up next

@cryptoAtwill
- Adding https://github.com/consensus-shipyard/builtin-actors/blob/master/primitives/src/taddress.rs to fvm-utils.
- Making all libraries modules of the crate exposed through lib.rs.
- Cleaning as much Filecoin-specific code as possible while still compiling (let me know if you need help with this)
- Adding in the README a brief explanation of the modules included in the crate and how to create a new sample actor using fvm-utils
@adlrocha
- Implementation of Mir-compatible full-nodes and syncing protocol for Mir.
- If finished before the end of the week, support Mir critical development.
@sergefdrv
- Look into the code in this PR on improving the atomic execution protocol and work further in that direction
@dnkolegov
- Start working on the integration tests for spacenet's Lotus
@matejpavlovic
- Mir State synchronization and starting from checkpoints.
- Trantor rebranding and documentation (more videos, CL Summit talk).

2022-10-25

✋ Attendees

@adlrocha
@dnkolegov
@matejpavlovic

📣 Updates

your notes here

🧵 Discussion

What we need for Spacenet deployment
- Formalize processes.
  - Restart from checkpoint.
  - Upgrading the network to a new version.
  - Tools (scripts) to do the above.
- Two deployment environments on AWS (staging deployment)
  - WAN (world-wide) - Great Bear :bear:
    - 1-2 bootstraps
    - 7 validators
    - Tutorial for anyone to become a learner in Spacenet.
  - WAN (regional) - Draco :dragon:
    - Same deployment as above, but in one geographical region (e.g. Europe)
  - LAN (local) - Scorpion :scorpion:
    - Same deployment as above, but in a single data center
- Website in scaling.filecoin.io/spacenet with tutorial and info.
  - Memberships (network addresses and public keys of nodes running the deployments)
  - Monitoring of SpaceNet
- Faucet for user funds.
- [Denis] Sanity-security check (black magic stuff).

2022-10-31

✋ Attendees

sync

📣 Updates

LabWeek ´22
@cryptoAtwill:
- Migrated hierarchical-sca repo to ipc-gateway with the following changes
  - use fvm_util runtime for the implementation
  - updated HCAddress to fvm_shared address v3+. In this case we are using delegated address type
  - clippy and code fmt
  - Created follow up issues(https://github.com/consensus-shipyard/ipc-gateway/issues)

🧵 Discussion

@adlrocha: @dnkolegov @matejpavlovic I am adding the following checklist to gather the list of tasks pending for SpaceNet launch so we can start assigning DRIs. Our focus these few weeks should be to "get ready for launch" :rocket:.
- [x] [DRI: @adlrocha/@dnkolegov] Finalize and test implementation of syncing protocol for Mir.
- [x] [DRI: @dnkolegov] Integration tests for Mir using itests.
- [x] [DRI: @dnkolegov] Vulnerability scanning and updating of Mir's dependencies.
- [x] [DRI: @dnkolegov] Vulnerability scanning and updating of Mir's and Lotus's dependencies.
- [x] [DRI: @dnkolegov] Run security linters against the Mir code.
- [x] [DRI: @dnkolegov] Run security linters against the target Lotus code.
- [x] [DRI: @dnkolegov] Run goroutine leak tests the Mir code.
- [ ] [DRI: @dnkolegov] Add integration tests from this doc.
- [ ] [DRI: @adlrocha] SpaceNet genesis and faucet strategy (determine how we are going to provide users with funds).
- ~~[x] [DRI: @matejpavlovic] Bug when a validator catches-up with the SMR and sends the first user message (bug found for chat-demo in Lisbon).~~
- [ ] [DRI: ] Document describing the different operational processes for SpaceNet (deployment, "catastrophic" restarts, upgrades, onboarding new validators, getting started for users, get funds, etc.).
- [ ] [DRI: @matejpavlovic] Infrastructure for "constellations" deployment and monitoring (Great Bear, Draco, Scorpion).
- [ ] [DRI: @dnkolegov] Perform basic securilty checks against the infrastructure.
- [ ] [DRI: @jsoares] Website and/or blog post announcing SpaceNet.
  - Plan is to get scaling.filecoin.io up and add network technical details to github (a la wallaby)
- [ ] [DRI: @all] Roadmap for SpaceNet (what can users expect to be able to do in SpaceNet in the next few months?)
- ~~[ ] [DRI: @matejpavlovic] Update mircat event inspector and debugger.~~
  - Taken by Alejandro later
- [ ] [DRI: @matejpavlovic] Set up machines for :bear: :dragon: :scorpion:
- [ ] [DRIs: see issues] Mir-level testing, see GitHub issues (DRIs to be assigned)
@dnkolegov:
- added several tasks for me to the list above

🎯 Up next

@adlrocha:
- Syncing protocol implemented and hand-off to Denis for integration tests.
- Support Willes with new IPC Gateway design decoupling functionality in different actors.
- Add discussion in FIP repo for IPC in preparation for core-devs call.
- TBD after assigning tasks.
@dnkolegov: do the assigned tasks
@sergefdrv: finalize discussions on atomic execution mechanisms and use-cases for IPC and start coding :keyboard:
@cryptoAtwill: migrate atmoic exection to its own repo, making sure it compiles to wasm.
@jsoares: set up website content

2022-11-07

✋ Attendees

async

📣 Updates

@adlrocha
- Implementation of syncing protocol and integration of itests for Mir.
  - PR: https://github.com/consensus-shipyard/lotus/pull/19
- Preparing core-devs meeting where we will present the consistent broadcast FIP, and IPC
  - Slides: https://docs.google.com/presentation/d/13dgXVpnDwkTNFtrKb-XaakYEl6I9rF_dpULcOvKCXhc/edit#slide=id.gae70f6dc90_0_962
- Working on bugs and ways to improve the forking rules in case of crash of a Mir validator.
- Review new user-defined IPC actors design and implementation and decoupling of the atomic execution logic into its own actor.
@cryptoAtwill
- Break ipc-atomic-execution into separate repo
- Expose ipc-gateway through feature and crates
- Started migrating ipc-subnet-actor
@sergefdrv
- LabWeek'22, Week 2
- More discussion on atomic execution mechanisms and use-cases for IPC
- Review of Atomic execution actor PR draft
@dnkolegov:
- working on itests for spacenet

🧵 Discussion

@adlrocha: @jsoares, I went through the planning and issues at consensus-shipyard/lotus and I would say that they are up to date, but let me know if you need some additional input regarding pllannin g and next steps, or if you want to do a quick sync to catch-up on this.

🎯 Up next

@adlrocha:
- Continue with testing and improving itests.
- Implement fork rules for mir-validator crashes.
- Start planning the implementation of the faucet and the genesis actor.
- Focus on having our first testing infra of Spacenet (:scorpion:)
@cryptoAtwill:
- Migrate ipc-subnet-actor completely
- Discuss what to do next:)
  - @adlrocha: Created this issue to track pending tasks: https://github.com/consensus-shipyard/ipc-gateway/issues/9
@sergefdrv:
- Remotely attending ConsensusDay + ACM CSS'22
- Continue working on improved atomic execution
dnkolegov:
- continue working on itests and bug fixing

2022-11-14

✋ Attendees

async

📣 Updates

@adlrocha
- Mir integration in Lotus finalized. We are wrapping up the last few basic itests from the testing doc before we "officially" push the PR to Lotus master, and we are planning the next batch of more "complex scenarios".
  - Upgraded to the latest version of Mir (fixing some networking bugs and with some performance improvements in the checkpoint protocol).
  - Initial discussions and plan to make Medusa the first use case to deploy over Spacenet.
- First strokes of the Spacenet faucet.
- First draft of the Spacenet/IPC refined roadmap. To be reviewed in this week's team CoLo. Feel free to add initial feedback and suggestions.
- There's been a lot of progress moving IPC actors to user-land (great job here, Willes!)
  - Housekeeping of issues and state of the code.
  - Tracking issue with pending tasks for a first release of the user-defined actors. With this we would be able to start with the integration with Lotus (and this will kick-off a hard conversation, as native actors are no longer supported in Lotus upstream :) yay!)
  - Subnet actor being refactored to use our fvm-utils runtime abstraction.
  - Atomic execution decoupled from core functionality of IPC gateway (this will allow custom user-defined implementation of atomic execution protocols that leverage the ipc-gateway. We just provide a reference implementation).
  - Proposal for actor interface identification (required for our protocol so the ipc-gateway is able to, for instance, identify if an actor implements the subnet actor interface).
- Presented Consistent Broadcast FIP and IPC in core-dev meetings. Some points of feedback we received:
  - For IPC, people were really interested in seeing a roadmap and understanding how it would change Filecoin in the future (the website, the video, and the roadmap we are working on should help with this).
  - Alex was really interested in the scope of IPC to work on a more general universally discoverable network identifier (analogous to the IPC address we currently provide as part of IPC). This is a line a work maybe worth exploring in the short-term.
@cryptoAtwill
- Migrate ipc-subnet-actor constructor and join method with test cases
- Migrate ipc-subnet-actor leave and kill methods
@dnkolegov:
- itests for Spacenet and Mir
- bug finding and debugging
@matejpavlovic:
- Remote deployment infrastructure and documentation for Mir. This automates installing and deploying mir on remote machines, given only their IP addresses and an SSH key. It can be used to run the dummy chat demo app and rudimentary benchmarks.
- Starting Trantor nodes from arbitrary checkpoints - Trantor now exposes periodic checkpoints that the application can store on disk. When starting a Trantor node, it can be told to load its state (including the state of the application running on top of it) from one of these checkpoints.
- Consolidated delivery of Trantor's "padding" blocks - even empty blocks resulting from leader failures and view changes are explicitly delivered to present a more simple and predictable behavior towards the application consuming them.
- CL Summit Talk: Slides, Video (Password: YV9EBZ^k)
- Talked to Joao Leitao and his students about combining efforts on Mir and Babel (code here). They seemed extremely enthusiastic in Lisbon, wanted to meet on-line to discuss further steps, but are not very responsive now. Will ping them again after a timeout.
- Client watermark implementation for Trantor - necessary for preventing infinite growth of memory footprint
- Finalized videos on Mir coding. They are the beginning of a bigger video tutorial on Mir. Producing further videos is on hold until Spacenet launch.
  - Mir Introduction: Principles of Distributed Programming, slides,
  - Mir Ping-Pong: Writing low-level code for a distributed protocol, slides
- New Mir project on interoperability - for, e.g., an internship
- Proposal for new Mir event interface
- The Trantor protocol - this document is describing Trantor as a protocol, created by transforming the "Eudico's ordering layer" doc. It should evolve to the form of a paper (officially published or not - different question). It's still WIP, the first cca. 2/3 should be quite readable, but still with a few TODOs and missing proper introduction.
- Started writing Spacenet deployment tools - still WIP, failing to run Lotus nodes on different machines
@sergefdrv
- Draft PR to support wrapped cross-net message delivery
- Implementing this proposal to improve atomic execution in IPC

🧵 Discussion

🎯 Up next

@adlrocha:
- Spacenet CoLo in Madrid.
- First long-running deployment and faucet finalized.
- Code-freeze and cut release.
@cryptoAtwill:
- Migrate ipc-subnet-actor completely
- ipc-gateway add more unit tests
@dnkolegov:
- Finalizing itests
- Code review for the Mir intergration PR
@jsoares
- Continue work on the website
@sergefdrv
- Continue implementing improved atomic execution scheme

2022-11-21

📣 Updates

@cryptoAtwill:
- fixed migration errors in leave and kill..
- Migrated ipc-subnet-actor completely, migrated submit-checkpoint and all its related code. Updated testcases to use MockedRuntime. Also added a few more test cases in submit-checkpoint.
- Proposed https://github.com/consensus-shipyard/ipc-subnet-actor/issues/10 for some simple code refactoring
@adlrocha
- Spacenet
  - Spacenet infrastructure deployment.
  - Fixed a nasty bug when validators restart and need to catch-up with the state of the chain. The bug involved a race in the Lotus integration, and a race in Mir. Now all tests pass 🎉 (thank you Matej for an amazing job here! It wasn't easy).
  - Re-architected Mir integration in Lotus to make it less race-prone.
  - Faucet frontend, and integration with peers and wallet key tested.
  - Spacenet README and getting started for users and developers.
  - Lots of progress for the first release of IPC actor implementation (1,2,3)
- Next steps for IPC after Spacenet discussion.
@jsoares
- Scaling Filecoin minisite making progress, but content still to be completed
- Spacenet faucet deployed (link above!)
- Helped track down connectivity issue :)
@dnkolegov
- Faucet
  - Added a basic implementation of the Faucet
  - Refactored the Faucet implementation using the well-established web service template from Ardan Labs
  - Added an integration test to the Faucet
- Shipyard-lotus
  - Debugging the Lotus-Mir race bug in the shipyard
  - Fixing and refactoring tests
  - Adapting CICD
@sergefdrv
- Adjusted and merged PR to improve the cross-net message mechanism in IPC – pass to the destination actor information about the origin of the cross-net message
- Working on implementing improved atomic execution scheme

🧵 Discussion

@adlrocha: @dnkolegov, as discussed in Slack I suggest focusing on any of these tasks next. Which one you prefer? Or is there anything that we have peding that you want to focus on?
- Include support for reconfiguration in Mir-Lotus to include support for dynamic membership in Spacenet.
- Start integrating all the code from Eudico to spawn new subnets (and re-architect it to avoid having to have all subnet process as go-routines within the same process --we already learnt how hard to handle that monolith was--).
- @dnkolegov:
  - What about testing the recovery mechanism or adding more tests to the shipyard? I am not sure we want adding more code before the luanch and release and I think we should be focusing on robustness and the quality of the current codebase.
- As discussed sync:
  - @dnkolegov will be focusing on bringing reconfiguration support to Spacenet and testing it.
  - @adlrocha will be working on catastrophic recovery and chain snapshots. @dnkolegov will give support testing this new feature.

🎯 Up next

@cryptoAtwill:
- Migrate ipc-gateway unit tests
- Integration testing of ipc-gateway and ipc-subnet-actor on local fvm.
@adlrocha
- Infrastructure deployment for Spacenet and final integration of the faucet with the backend.
- Bootstrap nodes deployment and configuration.
- Implementation of Spacenet catastrophic recovery support.
- Contribute to the scaling.filecoin website.
- Figure out next steps post-Spacenet.
@sergefdrv
- Continue implementing improved atomic execution scheme, possibly finish the coordinator part
@jsoares
- Get the website ready by EOW

consensus-shipyard / consensuslab