Create service quality test

gavv commented 4 years ago

Intro

Currently we have unit tests for internal components (src/tests) and a few simple integration tests for public API (src/tests/public_api).

All these tests are non-real-time, i.e. they don't expect any action, like delivering a packet, to take any specific amount of time. Actually they mostly are not tied to time at all. It has both advantages and disadvantages:

Tests can run in any conditions. We run them on CI, which workers are often slow, and on valgrind, which also randomly slows down the test execution.
On the other hand, such tests can't check how well Roc behaves in real-time conditions. E.g. what is the service quality and the latency, are there underruns and overruns, etc.

To fill this gap, we have created this repo, rt-tests. It will contain various real-time integration tests for public API.

This repo will contain non-functional tests, i.e. tests which result is not just OK/failed, but instead it may be some metric (e.g. 90% loss) or even a plot (e.g. latency trend). We will periodically run these tests on real hardware and analyze results for regressions. In future we will also run this tests on real networks.

Unlike the main repo, rt-tests uses CMake, C++17, and Google Test.

Task

Currently rt-tests contains a skeleton of ServiceQuality test.

https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp

This issue is for completing that test.

The ServiceQuality test should start two threads. The first thread should write some predefined sequence of samples to the sender. The second should read samples from receiver and check the sequence.

There should be several sub-tests.

First we should check that the sample sequence is the same on sender and receiver (except possible leading zeros) when there is no packet loss.

Then we should introduce some amount of packet loss. It can be done by implementing a simple UDP-level proxy, like we do in our current integration test (but without using internal roc modules). Then check what is the service quality on different combinations of: FEC scheme, FEC block size, FEC redundancy level, and network loss level. We should test three FEC schemes: none, Reed-Solomon, LDPC-staircase.

The service quality can be measured as the percent of dropped samples, i.e. samples that are equal to zero instead of the expected value (on receiver). 100% service quality on 5% packet loss means that all 5% of lost packets were restored using FEC. 95% service quality on 10% of packet loss means that we restored 5% of packets, but was not able to restore another 5% of packets.

We can print a table with results (measured service quality values for different combinations of parameters) to stdout.

In future we will extend this test to run over real network to measure actual service quality on different network types.

Checks

This issue provides an example of how you can produce samples on sender and check them on receiver: https://github.com/roc-streaming/roc-go/issues/101

That algorithm was implemented in this PR: https://github.com/roc-streaming/roc-go/pull/111

These issue and PR are from roc-go repo (Go bindings for roc-toolkit), where we have a test that has some similarities with the one described here.

Info

See "Workflow" section in README: https://github.com/roc-streaming/rt-tests#workflow

Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html

FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html

Our current non-real-time integration test can be found here: https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp

black-eagle17 commented 3 years ago

@gavv Greetings again. Since you mentioned we should not implement the UDP Proxy with roc internal modules. Should we be using some sort of third party library? Or can you go more in depth about the Proxy?

gavv commented 3 years ago

Hi, sorry for delay.

Yes, most reasonable is to use some cross platform library that allows to send and receive UDP packets. For example, libuv, libevent, POCO, Asio. I'd prefer something lightweight like libuv.

By UDP proxy I mean a simple thing that receives datagrams on one address and resends them to another address. For the purpose of testing it can drop, reorder, duplicate, or damage some of the datagrams.

roc-streaming / rt-tests