Experimental design

[SebastianElvis]

Problem

We should do experimental evaluation for RandChain.

Description

We can follow OHIE's experimental designs. We should test the following things:

• Block propagation delay (BPD): How long a block reaches the majority (e.g., 99%) of nodes?
• Confirmation latency: Hong long it takes to confirm a block (given a block time)
  • We may also need to compare this with other blockchains
• Decentralisation factor:
  • Hashrate between fast miner v.s. slow miner
  • Distribution of miners on the mined blocks

Alternatives

Additional context

OHIE paper: https://arxiv.org/abs/1811.12628 OHIE code: https://github.com/ivicanikolicsg/OHIE

Implementation

• [x] I would be interested in implementing this feature.

[SebastianElvis]

Specification of the experiment

Hardware

• AWS machine: t2.micro (see https://aws.amazon.com/ec2/instance-types/)
• Number of instances: 1000
• Regions: [ap-northeast-1, ap-northeast-2, ap-south-1, ap-southeast-1, ap-southeast-2, ca-central-1, eu-central-1, eu-west-1, eu-west-2, sa-east-1, us-east-1, us-east-2, us-west-1, us-west-2]

Software

OS: AWS basic AMI (CentOS) Rust version: 1.45.0 RandChain version: TBD Blocktime: 60s

Methodology

1. Run RandChain on all instances for 1 hour
2. Shutdown instances
3. Collect logs
4. Parse logs

Micro benchmarks

1. Solve/Prove/Verify efficiency

• Definition: time cost for running Solve/Prove/Verify
• Example: we already have it
• Expected result: Pie19 is best

2. CPU utilisation

• Definition: average used CPU when running SeqPoW
• Example: HydRand paper Appendix A
• Expected result: not fully occupying CPU -> energy efficient (rethink this word)

Marco benchmarks

1. Block propagation delay (BPD)

• Definition: how long it takes for a block to be propagated to the entire network?
• Example: OHIE paper Section 6A/6B
• Method: Collect when nodes receive the block, calculate time difference
• Expected result: much better than other protocols

2. Decentralisation factor

• How many different miners are generating blocks?
• Blocktime v.s. decentralisation factor

3. Network bandwidth utilisation

• Definition: average used bandwidth per node
• Example: HydRand paper Appendix A
• Method: On every node, run dstat
• Expected result: much better than other protocols

[ChrisLinn]

Method: Collect when nodes receive the block, calculate time difference

say a block A is rolled back to be block B, then block B is rolled back to be block A again,how do we test this case?

[SebastianElvis]

Method: Collect when nodes receive the block, calculate time difference

say a block A is rolled back to be block B, then block B is rolled back to be block A again,how do we test this case?

Rollback is in fact irrelevant to BPD. BPD only considers when a block is received, but does not consider when a block becomes stable.