Improve tests

[sezanzeb]

• Removing uinput_write_history and only using uinput_write_history_pipe
• Always using cleanup instead of quick_cleanup. Makes tests take longer, but maybe there can be optimizations
• helper functions for waiting for expected injected events that time out at some point.
• The fixtures are a bit odd to use, it usually involves using their path (like "/dev/input/event30") in some constructor. Instead, some constant would be good. Like GAMEPAD = "/dev/input/event30" or something similar, or maybe preparing an InputDevice that is ready to use for all tests for each fixture, which is reset in quick_cleanup
• moving test utilities from test.py into separate module

[jonasBoss]

I heave a few thoughts on this:

• create a tests.toolbox module.
  • we currently have small helper functions scattered all over the tests, they should go in there
  • move all global variables and functions from tests.test to the toolbox, that way we don't need the the workaround I implemented here: https://github.com/sezanzeb/input-remapper/pull/228/commits/da69aad60cc6025a3c644100669b389364ce92b0

• write toolbox functions for common data structures and method arguments. e.g.:

  • custom_mapping.change() has an extra argument in the independent_uinput branch. There are many places where I needed to change that, a simple toolbox method could have made that a lot less work.

    def custom_mapping_change_args(key=Key(EV_KEY, BTN_A, 1), target="keyboard", mapping="a"):
     return (key, target, mapping)
    
    custom_mapping.change(*custom_mapping_change_args(mapping="c"))

    now we only need to change the custom_mapping_change_args function and edit the few places where we need a gampad or a mouse as target.

• A failed test should only fail because:
  • we actually changed the behaviour of the code
  • a toolbox functions needs updating to account for a code refactor
• always think of a unit as a blackbox:
  • only test input and output to the unit
  • never rely on, or test code internal to the unit
  • a code refactor internal to the unit can not brake the test, unless there is actually a bug with the refactor
• The largest unit is the whole program
  • a few test which cover common use-cases for the whole program will probably cover the majority of the code.
  • tests which target the whole thing and not a specific module or function will never brake (assuming the inputs and outputs stay functionally the same) even something that changes the input and output (independent_uinput) should not actually brake those tests, because it does not really change the behavior observed by a user. It will obviously brake the testing, but not the test itself, only toolbox functions, that need to be adapted to account for the changed uinputs
  • only write additional tests for smaller units if they are not completely covered by a test to a larger unit.

[LunNova]

It'd be nice to have a set of tests that are safe to run on CI and don't depend on system state to run. (The ones which are truly unit tests?)

Some tests care about timing (may fail if build machine is slow), some care about system services or start external processes, and some need a GUI.

I've been trying to package this for nixos and this is the set of tests I've ended up with that seem to be okay for CI:

https://github.com/NixOS/nixpkgs/pull/155290/files#diff-abc835f871a20a77507a0632a2791efaa1d0498ce56ca7b3c88f38a64b36b990R83

[sezanzeb]

It then probably makes sense to split them into two directories, "integration" and "unit". Can do

[sezanzeb]

@jonasBoss are tests from test_gui super laggy for you out of nowhere as well? python3 tests/test.py integration.test_gui

even on the main branch

[sezanzeb]

the commit hash that introduces the lags is 47bcefa, no lags on 76c3cadcfaa3f244d62bd911aa6642a86233ffa0. ffs

[jonasBoss]

Seems fine to me. python3 tests/test.py integration.test_gui runs in 25s

[sezanzeb]

self.get("code_editor").get_buffer().set_text(symbol or "") causes it. I can't believe it. Started to happen out of nowhere!

everything freezes up on my computer until the tests are finished if this line is executed. I'm going to do a full system upgrade now and see if that fixes anything

[sezanzeb]

Tests should probably be very loose on timing requirements from now on.

They should also not use asyncio sleeps to wait until a macro is done or whatever. Instead they should wait for the coroutine to finish.

[jonasBoss]

Just stumbled upon this. I think it provides valuable ideas on how we can improve and simplify our test: https://www.youtube.com/watch?v=Bq_oz7nCNUA

[sezanzeb]

@jonasBoss are tests from test_gui super laggy for you out of nowhere as well? python3 tests/test.py integration.test_gui

even on the main branch

Might have been because of a broken/bloated ssd, or maybe because of an update to dependencies in the meantime. They are not laggy anymore for me.