search

status-im / swarms

Swarm Home. New, completed and in-progress features for Status
92 stars 31 forks source link

Extended E2E tests for message reliability #91

Closed mandrigin closed 6 years ago

mandrigin commented 6 years ago

Preamble

Idea: #91
Title: Extended E2E tests for message reliability
Status: Draft
Created: 2018-03-12
Requires (*optional): <Idea number(s)>
Replaces (*optional): <Idea number(s)>

Summary

After discussions, we decided will keep 2 tracks there:

TRACK NO. 1 — Networking and message delivery

Use status-go and Docker compose (with networking capabilities) to create extended automated end-to-end tests for different real-life scenarios of chat message delivery.

There are 3 main dimensions we want to cover with these tests: 1) Test Scenarios 2) Network Topology 3) Network Conditions

Each (1)-(2)-(3) represents a single test (e.g. "1-1 chat(1) in the star topology with a mailserver(2) in the excellent network conditions(3)").

What tools will we need?

(1): auto-testing framework, headless Status (aka "Status CLI") (2): docker-compose (3): https://github.com/worstcase/blockade or https://hackernoon.com/network-emulation-for-docker-containers-f4d36b656cc3

What metrics do we check there?

Test Scenarios

a. 1-1 chats (Alice, Bob)

  1. Alice, Bob online. Alice messages Bob, Bob messages Alice
  2. Alice online→online, Bob offline→online. Alice messages Bob. Alice is online. Bob comes online.
  3. Alice online→offline, Bob offline→online. Alice messages Bob. Alice goes offline. Bob comes online.

    b. Open groups

  4. Message to the group.
  5. Receiving history of messages for the group.

    c. Closed groups

  6. Message to the group.
  7. Receiving history of messages for the group.

    d. Stress-test scenarios

    TBD

Network topologies

a. "Normal" scenario (mailserver, bootnodes)

  1. Alice and Bob are direct peers
  2. Alice and Bob aren't direct peers

b. No mailserver

  1. Alice and Bob are direct peers
  2. Alice and Bob aren't direct peers

c. Edge case topologies

  1. A chain: Alice and Bob are connected through N nodes, no mailserver.

Network Conditions

Excellent

Normal

Bad

TRACK NO. 2 — finding rendering issues and bottlenecks

The purpose of this track is to stress UI and find rendering issues when receiving a lot of messages. These tests could be relatively short and simple. status-go-bots could be used to generate N messages within M seconds for K keys. Then the Android client would receive them and see if there are no issues rendering them. The goal is to have multiple test cases tested nightly in Jenkins. These tests will either run on an Android Emulator or on some mobile test cloud.

What tools do we need?

What metrics do we check there?

Swarm Participants

Product Overview

Product Description

Requirements

  1. Logs
  2. status-go updates. Tests should run on the latest status-go.
  3. Timing requirements. Tests should be able to run overnight (~7-8 hours). Nightly jobs on Jenkins.
  4. Measuring timing threshold.
  5. Statistics dashboard (Grafana or CI dashboard #90)
  6. Running on a private network.

Useful links & Dependencies

Minimum Viable Product

TRACK NO. 1 Goal Date: 2018-04-12 Description: Make a job on Jenkins that runs docker-compose tests for sending/receiving messages nightly.

TRACK NO. 2 Goal Date: 2018-04-12 Description: Make a job on Jenkins that runs a simple public chat scenario nightly. Report if succeeds/fails. It would be nice to test both develop and the new protocol branch.

GitHub project

https://github.com/orgs/status-im/projects/20?add_cards_query=is%3Aopen

Dates

Goal Date:

Description:

Testing Days required:

Success Metrics

KR: sent/delivered ratio is ~ 99%

Exit criteria

  1. All the testing scenarios are implemented and are running on Jenkins
  2. All the obvious issues with sending/delivering messages are identified and fixed

Supporting Role Communication

Copyright

Copyright and related rights waived via CC0.

adambabik commented 6 years ago

Alice online→online, Bob offline→online. Alice messages Bob. Alice is online. Bob comes online. Alice online→offline, Bob offline→online. Alice messages Bob. Alice goes offline. Bob comes online.

I believe in these two cases the important part is to test requesting historic messages, right?

Alice and Bob are direct peers

This is never a case in the real world. Two Status apps can't be connected directly. (This is unknown when we decide to use discovery protocol, but we should not worry about it here. Let's use the current possible topologies.)

mandrigin commented 6 years ago

@adambabik indeed.

dshulyak commented 6 years ago

btw mvp is implemented in our regular e2e tests. i would extend this mvp to send 10 messages to 10 different users :)

mandrigin commented 6 years ago

@dshulyak makes sense, though there is a difference: we will hopefully use a headless react client to really-really simulate real-world scenario (not only whisper delivery, who knows where the message could be lost). I'll update the MVP with that. That and also to set up an environment, regular CI runs, etc is enough of a task, IMO. After that is done, we can scale our tests to something more exciting.

lukaszfryc commented 6 years ago

@mandrigin 1st track will tests messages in different network conditions. Will it be possible to have test environment with different topologies for manual tests as well? I'd really like to test how the app behaves (from user point of view) when part of the infrastructure is offline or timeouts. For instance when Alice and Bob are connected through N nodes, no mailserver.

I was thinking about having different topologies in docker compose and then using this network by the app.

oskarth commented 6 years ago

Closing this issue as part of spring cleaning. If this idea is still relevant, please submit a PR per https://github.com/status-im/ideas/#contributing. If you feel closing this issue is a mistake, feel free to re-open.