Extend end-to-end test framework to integrate traffic simulation

[kschiffer]

Summary

To extend our end-to-end tests with test-cases that comprise network data, we need to extend our framework there to be able to simulate network traffic easily.

Why do we need this?

• Cover critical network logic with end-to-end tests
• Enable non-frontend devs to write end-to-end tests

What is already there? What do you see now?

Cypress-based end-to-end test framework, which we currently use to write test cases centered around the frontend.

What is missing? What do you want to see?

Some plugins or added cypress commands to inject network traffic into the stack. E.g. to be able to test join procedures, uplinks/downlinks, message en/decoding, mac commands, etc.

How do you propose to implement this?

Our CLI already allows simulating network traffic. We can use this to inject network traffic from within tests and subsequently check on expected behavior in the console (via cy.exec()). Since Cypress wraps an assertion library, we can also test CLI commands and their output directly.

Can you do this yourself and submit a Pull Request?

@bafonins or me to set up the groundwork.

It would be good to know from everybody which example test cases you'd like to add and whether CLI based traffic simulation and asserting on CLI command output and/or frontend behavior are sufficient to achieve these. cc @rvolosatovs @KrishnaIyer

[kschiffer]

Specifically, what I wonder is whether these functionalities will suffice for our end-to-end tests:

• [ ] Injecting traffic (using TTS CLI)
• [ ] Running CLI commands (or any arbitrary executable)
• [x] Making API requests
• [x] Perform frontend interactions
• [x] Perform database manipulation #3605
• [ ] Asserting on the output or stream of CLI commands (or any arbitrary executable)
• [x] Asserting on the behavior of the frontend
• [x] Asserting on the result of API requests
• [x] Asserting on database content #3605

[nejraselimovic]

@kschiffer do you still need input from others to proceed with this?

[johanstokking]

@KrishnaIyer can you book a few days to get this going?

[KrishnaIyer]

I'm handling this already with https://github.com/TheThingsIndustries/lorawan-stack/issues/2742

[kschiffer]

I think this has been closed in error. While we have traffic simulation tests, these are not facilitated through the Console. I'd like to add a couple of tests that check whether traffic is being handled correctly in the Console.

This includes:

• Testing the event stream
  • Event filters
  • Data inspector
  • Preview row
• Testing "last activity" updates
• Test "Device joined successfully" message when a device in an application has joined the network
• Testing locations set by uplinks (gateways and devices)