This PR contributes a new architecture to OpenCBDC currently named "3PC" (due to an additional phase over 2PC). Unlike the two existing architectures (atomizer and 2PC) which assumed a UTXO model compatible with a UHS, 3PC makes no assumptions about the underlying data model or transaction semantics. Instead, 3PC implements a strongly-consistent, fault-tolerant, concurrency controlled, generic key-value store. Specific transaction logic and data representation is implemented by pluggable modules called "runners" which only affect a single component of the overall system. This makes the 3PC architecture extremely flexible for rapid prototyping, supporting arbitrary programmability, transaction logic or data models without having to modify core system components. It can even support multiple transaction formats simultaneously, including a mixture of UTXO and account-based data models. It would be fairly trivial to implement a runner which executes the transaction format used in 2PC/Atomizer.
There are two runners provided in this PR. The first is a fairly generic function execution environment using the programming language Lua (called "3PC/Lua"). The transaction semantics and data model are implemented as Lua contracts and thus can be modified while the system is running (and transactions are in-flight) without modifying any C++ code. An example contract is provided with implements a simple account-based transaction format and data model. A benchmarking tool and wallet is also provided as a usage example.
The second runner implements the Ethereum Virtual Machine (EVM) and can execute raw Ethereum transactions verbatim (called "3PC/EVM"). It provides an HTTP JSON-RPC server which mimics the Infura/Geth RPC interface, enabling Ethereum tooling to be used out-of-the-box in many cases with 3PC/EVM. We provide a benchmarking tool for testing base token transfers and transactions calling an ERC20 contract. Unlike in Ethereum, transactions in 3PC/EVM execute in parallel rather than being sequenced by blocks, but 3PC enforces concurrency control resulting in equivalent transaction semantics. Transactions are guaranteed to complete eventually if there are no system failures, even when multiple transactions conflict with each other.
Design
Similarly to Atomizer and 2PC, data is partitioned into shards. In 3PC, shards support multiple rounds of locking for a single transaction using two-phase locking, and shards enforce concurrency control by making transactions queue for locks. Each transaction is assigned a monotonically increasing "ticket number" by a replicated service called the ticket machine. The shards use the ticket numbers to implement the "wound-wait" deadlock avoidance algorithm which prioritizes fairness over efficiency. Transactions are coordinated between shards by an "agent" which contains both a transaction coordinator (similar to 2PC) and the contract runner itself. Unlike in 2PC, the coordinator/agent is not replicated, only shards and the ticket machine are raft replicated services. Instead, we introduce an additional commit step to the shards to enable agent recovery after a failure. Similarly to 2PC and unlike Atomizer, 3PC can scale horizontally by adding more shards and agents. Tickets are issued by the ticket machine in batches, the size of which can be increased significantly to alleviate the bottleneck at the ticket machine.
Performance
We benchmarked 3PC/EVM replicated in three AWS regions (Virginia, Ohio, Oregon). We used the example ERC20 contract from OpenZeppelin and our benchmark repeatedly sent between paired accounts in the contract. With 128 shards and 64 agents we were able to achieve a peak throughput of ~140K TX/s with 99.999% of transactions finalized in under 4 seconds.
Future work
Experiment with and understand how lock contention affects 3PC performance
Address various TODOs in the code, some of these are related to things like Raft snapshots (which are not yet implemented), and others are just ideas for future improvements
Finish implementing the Geth RPC interface in the agent. This was previously implemented almost completely in a Python program (not included in this PR), but turned out to be too slow once benchmarked.
More tests (system, integration and unit)
More documentation/usage examples
More runner implementations (different contract languages, transaction formats etc)
Interoperability between runners
Shard data rebalancing (using a directory service managed by the 3PC protocol)
metalicjames
commented
1 year ago
Intro
This PR contributes a new architecture to OpenCBDC currently named "3PC" (due to an additional phase over 2PC). Unlike the two existing architectures (atomizer and 2PC) which assumed a UTXO model compatible with a UHS, 3PC makes no assumptions about the underlying data model or transaction semantics. Instead, 3PC implements a strongly-consistent, fault-tolerant, concurrency controlled, generic key-value store. Specific transaction logic and data representation is implemented by pluggable modules called "runners" which only affect a single component of the overall system. This makes the 3PC architecture extremely flexible for rapid prototyping, supporting arbitrary programmability, transaction logic or data models without having to modify core system components. It can even support multiple transaction formats simultaneously, including a mixture of UTXO and account-based data models. It would be fairly trivial to implement a runner which executes the transaction format used in 2PC/Atomizer.
There are two runners provided in this PR. The first is a fairly generic function execution environment using the programming language Lua (called "3PC/Lua"). The transaction semantics and data model are implemented as Lua contracts and thus can be modified while the system is running (and transactions are in-flight) without modifying any C++ code. An example contract is provided with implements a simple account-based transaction format and data model. A benchmarking tool and wallet is also provided as a usage example.
The second runner implements the Ethereum Virtual Machine (EVM) and can execute raw Ethereum transactions verbatim (called "3PC/EVM"). It provides an HTTP JSON-RPC server which mimics the Infura/Geth RPC interface, enabling Ethereum tooling to be used out-of-the-box in many cases with 3PC/EVM. We provide a benchmarking tool for testing base token transfers and transactions calling an ERC20 contract. Unlike in Ethereum, transactions in 3PC/EVM execute in parallel rather than being sequenced by blocks, but 3PC enforces concurrency control resulting in equivalent transaction semantics. Transactions are guaranteed to complete eventually if there are no system failures, even when multiple transactions conflict with each other.
Design
Similarly to Atomizer and 2PC, data is partitioned into shards. In 3PC, shards support multiple rounds of locking for a single transaction using two-phase locking, and shards enforce concurrency control by making transactions queue for locks. Each transaction is assigned a monotonically increasing "ticket number" by a replicated service called the ticket machine. The shards use the ticket numbers to implement the "wound-wait" deadlock avoidance algorithm which prioritizes fairness over efficiency. Transactions are coordinated between shards by an "agent" which contains both a transaction coordinator (similar to 2PC) and the contract runner itself. Unlike in 2PC, the coordinator/agent is not replicated, only shards and the ticket machine are raft replicated services. Instead, we introduce an additional commit step to the shards to enable agent recovery after a failure. Similarly to 2PC and unlike Atomizer, 3PC can scale horizontally by adding more shards and agents. Tickets are issued by the ticket machine in batches, the size of which can be increased significantly to alleviate the bottleneck at the ticket machine.
Performance
We benchmarked 3PC/EVM replicated in three AWS regions (Virginia, Ohio, Oregon). We used the example ERC20 contract from OpenZeppelin and our benchmark repeatedly sent between paired accounts in the contract. With 128 shards and 64 agents we were able to achieve a peak throughput of ~140K TX/s with 99.999% of transactions finalized in under 4 seconds.
Future work