[E2E] Byzantine network simulator environment

[dylanlott]

Objective

Make LocalNet be able to access arbitrary configurations to spin up any complex network. Nodes should be able to be fed arbitrary commands to act as different kinds of actors in the system. This should be plumbed with metrics for performance monitoring. The amount of overlap with this and the E2E test harness is left up to a spike.

Stress & Chaos testing are likely under the umbrella of Byzantine environments. This ticket should also explore adding chaos support for our Kubernetes environment.

Origin Document

466 specifically cut network simulation out of scope. This ticket addresses that scope to keep it tightly defined and well-documented.

Goals

• Maintain E2E suite passing
• Smoothly integrate with DevNet (Heterogeneous networks)
• Allow for the creation of complex network situations and the control of Byzantine actors.
• Explore chaos testing integration (possibly something like https://chaos-mesh.org/)
• Define any relevant metrics for performance monitoring

Deliverables

• [x] https://github.com/pokt-network/pocket/pull/664
• [x] https://github.com/pokt-network/pocket/pull/658
• [x] https://github.com/pokt-network/pocket/pull/699/
• [x] https://github.com/pokt-network/pocket/pull/702
• [x] https://github.com/pokt-network/pocket/issues/723
• [ ] https://github.com/pokt-network/pocket/issues/724
• [ ] https://github.com/pokt-network/pocket/issues/725
• [x] https://github.com/pokt-network/pocket/issues/726

Non-goals / Non-deliverables

• Change how LocalNet fundamentally works
• Break DevNet integration

General issue deliverables

• [ ] Update the appropriate CHANGELOG(s)
• [ ] Update any relevant local/global README(s)
• [ ] Update relevant source code tree explanations
• [ ] Add or update any relevant or supporting mermaid diagrams

Testing Methodology

• [ ] All tests: make test_all
• [ ] LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md

---

Creator: @dylanlott Co-owner: @okdas

[Olshansk]

@dylanlott I made some NITs to the issue description but was also hoping you clarify the goals & deliverables, perhaps with an example?

It's unclear to me exactly what the deliverable here is? New tooling? Documentation? Infrastructure? Should @okdas be a co-owner?

We can remove the triage label once we answer to above.

[dylanlott]

@Olshansk I'm spending time today shaping the actual deliverables for this issue based on my current experiments with LocalNet & Tilt. I think it's a good idea to make @okdas a co-owner on this, since it will be important to keep this in sync with DevNet as well as LocalNet.

[jessicadaugherty]

Hey @dylanlott! Trying to finesse this parent ticket and identify any child tickets that are required. Reading the proposed implementation here, these are potential tickets I noted that could live under the "deliverables" section of this ticket. Wdyt?

• Extend Debug CLI with new commands to control Byzantine actors
• Create Helm charts and Kubernetes manifests for Byzantine nodes
• Implement Chaos testing integration (e.g., using Chaos Mesh)
• Develop a mechanism for cycling nodes on and off to simulate leaving and joining the network (?)

ETA: lmk if the in flight tickets being here doesn't make sense

[dylanlott]

That looks good for deliverables, yes! The ticket might also want to mention #718 as possibly supporting some of these deliverables. If I come across any others I will link them back to this ticket as well.

[jessicadaugherty]

That would be great! Tysm @dylanlott I'll get started on those smaller tickets and add the links here.