🚧 | Y5: Implement existing BFT consensus in subnets

consensus-shipyard / consensuslab

The go-to place for scalable decentralised consensus research

65 stars 3 forks source link

🚧 | Y5: Implement existing BFT consensus in subnets #14

Closed jsoares closed 2 years ago

jsoares commented 2 years ago

Description

We intend to rapidly experiment with the hierarchical consensus architecture by integrating an existing, production-ready BFT implementation as a possible subnet consensus protocol. After analysing several options, we decided on Tendermint as a proof-of-concept.

This project will, therefore, consist of implementing, documenting, and evaluating Tendermint as a consensus layer for Eudico subnets. Tendermint Core will be used in the service-based model, rather than in the traditional ABCI-based model.

In our approach, Tendermint Core acts as a message system providing BFT total order broadcast of input messages without any modifications of the Filecoin state. Each Eudico node interacts with a trusted sidecar Tendermint node A Eudico node sends input messages to the corresponding Tendermint node; Tendermint validates them and ensures that the messages are recorded on every Tendermint node in the same order. The Eudico node then retrieves Tendermint blocks, perform static and semantic validation, and generates a Filecoin block with the verified messages, which can be applied directly.

Scope

Explore and test the expressiveness and power of Eudico’s design and interfaces (e.g., Consensus interface) by adding support for the Tendermint consensus protocol.
Improve Eudico characteristics by adding a BFT-type consensus protocol with better characteristics and justified properties in addition to already i plemented and used consensus protocols (e.g., PoW).
Research Eudico's “consensus upgradeability” and improve it if necessary.
Explore Filecoin design and Eudico implementation pitfalls for (potential) future integration with MirBFT protocols.

Dependencies

Resources

Talks

2022-02-07 ConsensusLab Team Week 22Q1: Y5 update

Demos

2022-02-10 Y5 Tendermint subnet (Denis)

2022-02-21 Y5 Tendermint fault injection (Denis)

2022-02-28 Y5 Tendermint fault injection (Denis)

jsoares commented 2 years ago

2022-02-21 Meeting Notes

Updates

Deployed a Tendermint testnet with 4 nodes
Found a simple way to add a new node to Tendermint testned, but! It also should be supported on the app layer. That will be the next step if necessary
Improved the implementation: found a bad cache that should support Tendermint connections
Performed two early internal demos: Tendermint in subnet, Tendermint in root net with fault injection
Merged the main eudico/filecoin branches

Discussion

How do the state transfers of Eudico and Tendermint cooperate?
Does someone need to have exclusive control?
Eudico block exchange:
Exchange block headers
Check in Tendermint whether it has been verified
Pick up the messages and check inclusion
Are we wasting bandwidth? It seems exchange is taking place at multiple levels.
This only happens during catch-up — otherwise you get it through gossipsub once.
[...]
Not a fundamental issue — implementation of MirBFT will sidestep it.

Up next

Understand how to use itests and start using it
Implement basic integration tests. We will get an answer to the question is it possible to use itests for consensus testing
Implement basic integration tests
Make a PR

jsoares commented 2 years ago

2022-02-28

✋ Attendees

@dnkolegov
@adlrocha
@matejpavlovic
@jsoares
@LefKok
@vukolic

📣 Updates

New demo: 4 Tendermint nodes (including 1 faulty node), 4 Eudico nodes, fault injection, root subnet with Tendermint consensus
Created a README document for Tendermint implementation
Created a checklist for a new consensus protocol implementation in Eudico
Still trying to adapt and run itests for Eudico, can't catch a bug, asked Alfonso for help

🧵 Discussion

Timeline recap
- Main blocker right now is itests. If we can get this right, it will also work for future consensus protocols and allow us to write different kinds of tests (inc. benchmarks).
  - Alfonso aiming to unblock this by tomorrow. Will have sync call.
- Planning to ramp up work on Y3. There are synergies between the two, inc. the upcoming testing work. Let's make sure the timeline for Y5 does not slide.
Consensus as a service
- Different number of Eudico nodes and ordering service nodes source of confusion in the past. Would it make sense to do this intentionally?
- What is a subnet? What is an Eudico node?
  - One could define a subnet as using an external ordering service. Whoever joins the subnet would implicitly trust external orderer.
- Endorsers and orderers in Hyperledger Fabric
  - Incentives?
- Blockchain bridge integration in our design? E.g. Tendermint node is Cosmos network. Cryptoeconomic model.
  - We also discussed having consensusless subnets in the past. Having a bridge just means some of the transactions coming from other networks.
  - Aspirational. We should have a design that doesn't preclude this but not an immediate goal.
  - Running MirBFT node as a separate protocol makes more sense in this model.
    - What's the MirBFT node identity?
    - Counterpoint: Shipping two binaries comes with operational issues and is also more resource-intensive. It would be nice to have a single one.
    - For subnets, we're doing everything as goroutines in the same process.
    - Avoid duplicating stack as we're doing with Tendermint

🎯 Up next

The main goal is run itests for Eudico

jsoares commented 2 years ago

2022-03-07

✋ Attendees

@adlrocha
@dnkolegov
@jsoares
@matejpavlovic
@vukolic

📣 Updates

Worked on itests
Added a basic test for Eudico PoW consensus
Added a basic test for Eudico delegated consensus

🧵 Discussion

How do we stand on reconfiguration?
- Does the Tendermint implementation support joining/leaving subnets?
- Right now, the node set-up is statically configured. We don't join or leave networks.
- But we support Tendermint as both root and subnet consensus (see early February demo), even simultaneously.
- This isn't a requirement for Y3 completion, but we'll need it for Mir.

🎯 Up next

Add a test for Tendermint
Clean Eudico itests code
Continue adapting itests to Eudico

jsoares commented 2 years ago

2022-03-14

✋ Attendees

@adlrocha
@dnkolegov
@jsoares
@matejpavlovic
@vukolic

📣 Updates

Created the PR for Tendermint code
Created two issues related to Consensus interface and delegated consensus block delay
Scripts and tests don't need Tendermint repo anymore

🧵 Discussion

Alfonso's found an issue:

We support the use of the same Tendermint blockchain by several subnets, right? And here we are registering a new subnet to the consensus. The assumption here is that only the owner of the node can tailor this registration message, right? Why I ask is because I am wondering how critical would it be for a rogue node to register a subnet he is not part of (or that doesn't even exist yet).

Yes, you are right. We register a Tendermint blockchain offset for a new subnet. Suppose we ran a Tendermint blockchain and a root subnet, then root's subnet offset may be 3 blocks. And then, after an hour, we decided to register a new subnet, its offset is 1000. If we don't use offsets then the second subnet will have to access all previous Tendermint blocks. So, yes, it is critical. A rogue node can register a network earlier than it would be registered, e.g. at epoch h. Then when the network is instantiated by real participants at epoch H they will have to process all blocks from epoch h to epoch H.
Way forward
- We're missing
  - decoupling subnets from Tendermint network (i.e., one Tendermint network/instance per Eudico subnet)
  - refactoring
  - benchmarking
  - misc fixes
- We're on track for EOM
- Non-Y5 items on Denis's plate (not critical for EoM, nice to have)
  - B1 itests for subnets
  - Y3 interfaces

🎯 Up next

Fix issues Alfonso has found, refactor the code according to the comments, test it

jsoares commented 2 years ago

2022-03-21

✋ Attendees

@adlrocha
@dnkolegov
@jsoares
@LefKok
@matejpavlovic
@vukolic

📣 Updates

Removed unnecesarry features from the codebase
Added acceptance tests to the README
Implemented a benchmark tool
Merged Tandermint consensus PR
Did code review
Added 3 issues and fixed 1 issue

🧵 Discussion

We can write a guest post in the "Decentralized Thoughts" blog
- Marko reaching out
- Focus: hierarchical consensus; start from paper and research blog, make it more technical
What kind of demo should I provide? Offline? What scenarios should I demonstrate?
- April: Eudico Garden, cross-subnet transaction
- Benchmark plan dependent on availability of tooling (pending Y2 outcome)
  - We could probably use traditional load generation tools.
  - We need to check whether API communication becomes the bottleneck (saturate API instead of consensus)
  - Demo benchmarking in May (go beyond Tendermint)

🎯 Up next

Implementing itests for subnet
Demo

jsoares commented 2 years ago

2022-03-28

✋ Attendees

@dnkolegov
@jsoares
@matejpavlovic
@vukolic

📣 Updates

Implemented a basic flow (happy path) covering the most of subnet features for different consensus pairs for one Eudico host: pow-pow, pow-tendermint, delegated-tendermint, etc.
Found several bugs, added several improvements.

🎯 Up next

Adapt tests for deleg-deleg and filcns-* consensus pairs
Look at Tendermint setup in Eudico-Garden

jsoares commented 2 years ago

The project wrapped up with the implementation demoed in the initial B3 project demo. See respective issue for more details: https://github.com/protocol/ConsensusLab/issues/6.

🥳