search

waku-org / pm

Project management, admin, misc
3 stars 1 forks source link

Waku Network Testing #2

Open jm-clius opened 2 years ago

jm-clius commented 2 years ago

Background

Waku network simulation and testing is done in collaboration with Kurtosis and uses their distributed network manipulation tools. The main aim of network testing is twofold:

The side effect is the creation of an integrated, multi-client simulation environment that can be used for regular integration testing, regression checks, network experiments, etc.

This issue tracks the main simulation scenarios we want to create and desirable outputs/metrics.

NOTE: This issue is WIP and will be adapted as more requirements from platforms materialise and as we familiarise ourselves with Kurtosis features.

First network test: relay scalability and familiarising ourselves with Kurtosis tools

The rough idea for a first test comes from this forum thread. It asks at least two questions about the simplest possible network setup, running only the main routing protocol (relay) and using only the nwaku client:

  1. at exactly what messaging load performance, latency and reliability starts to degrade or fall below an acceptable threshold
  2. at what network sizes performance, latency and reliability starts to degrade or fall below an acceptable threshold

Other metrics (such as node resource consumption - CPU, bandwidth and memory usage) should form part of the experimental output data.

Setup

Run a relay network with x amount of nwaku nodes using discv5 to establish a well-connected mesh (afaik this approximates a typical Communities setup on Desktop clients). Publish (from a random subset of nodes) messages of size s at a rate r.

Measure

Varying x, s and r, measure:

To measure reliability we can add a simple seq counter to the message payload. Latency can already be measured using the sender timestamp field.

Laundry list of scenarios we want to build

The scenario above proposes a way to get started with Kurtosis for network testing. The list below, captures more clearly the type of outputs that we want from network testing in the longer term:

It would be useful to come up with a set of automated e2e tests that can regularly run against an integrated simulated environment to detect regressions and performance issue early.

jm-clius commented 2 years ago

Issue copied over from https://github.com/vacp2p/research/issues/142

jm-clius commented 2 years ago

Original comment from @fryorcraken:

The side effect is the creation of an integrated, multi-client simulation environment that can be used for regular integration testing, regression checks, network experiments, etc.

I would call it a phase 2 more than a side effect.

I agree that priority is on multi-client scalability of Waku, followed by extracting performance metrics.

Phase 2 tests

What we would also like to have mid-term.

Non-regression testing: delta analysis

What we also need is various non-regression tests when (before) releasing new software. Where we do a run and compare with a previous run:

  1. Metrics: Extract response times, CPU Usage, etc (see metrics laundry list above). Any performance degradation? If so, investigate. If performance improvement, good to note and highlight.
  2. Errors: Check logs across all clients, any errors returned? Any new errors returned?

Multi-client functional testing

What we also need is a multi-client non-regression functional testing. ie, suite of functional tests that demonstrate cross client compatibility of several protocols. Similar to what is done in js-waku CI: [1] [2].

Does this fall under the Kurtosis umbrella?

jm-clius commented 2 years ago

Original comment from @fryorcraken:

Regarding phase 1, proving scalability. I suggest we focus on the Status client use case at first which enables to cover a good number of scenarios. This would also help us focus on answering questions relevant to a specific use cases and have some idea on what should be the scales of x, s and r.

We can structure the tests around answering this question: Can the Waku network onboard 10 million users?

We can start progressively by dialing n: 1,000, 10,000, 1mil, 10mil users. We can also identify a cap for a simulated environment and extrapolate from there (.e.g. maybe 10,000 is the max we can do with Kurtosis?).

We can start with topology similar to the expected one from Status clients:

Where # of go-waku nodes + # of js-waku nodes = n

From this simulation, we can extract/extrapolate:

corpetty commented 2 years ago

I've created a section in the Nomos tech roadmap that also details this work and tracks things, which can be viewed here.

You'll also find a document that I started to map out the detail the explicit differences between Status clients (Agents) w.r.t. what protocols they implement and their associated Waku messaging on the network. This is so that we can more realistically simulate the Status network during these tests. This WIP document can be found in the first link of the previously mentioned roadmap document.

alrevuelta commented 1 year ago

As a starting point to get started with Kurtosis and close the knowledge gap, @AlbertoSoutullo is willing to provide a set of configuration files to launch a simple simulation where:

The main goal is to use this simple setup as a starting point and the build on top to eventually end up with the First network test described here.

cc @LNSD @Daimakaimura

Ref: https://github.com/logos-co/wakurtosis

alrevuelta commented 1 year ago

As a continuation of my latest comment and as per the discussions with @AlbertoSoutullo @Daimakaimura, we have broken down into lower level requirements/tasks the First Network Test and its progress can be tracked here: https://github.com/logos-co/wakurtosis/issues/7