search

lcpz / lain

Awesome WM complements
GNU General Public License v2.0
1.05k stars 212 forks source link

We need a test suite #527

Open lcpz opened 2 years ago

lcpz commented 2 years ago

The idea is to have a lain-test.lua, which requires lain-test-{util, widget, layout}.lua, each of which requires all elements of the respective submodule with all possible options set. Tests could be performed with Xephyr as follows:

Xephyr :1 -screen 1280x800 2> stdout.txt & DISPLAY=:1 awesome -c lain-test.lua

This structure and workflow will serve as a unit, integration and system test suite.

They will also be used to do regression testing for each new commit (e.g., using Travis CI).

thomascft commented 2 years ago

I think rather than using xorg-server-xephyr we should use xorg-server-xvfb. Here's a dockerfile I spun up for testing just to figure out the concept. I'll modify the rc.lua file a bit for testing and setup Github Actions. I'll make a push when I get it working.

thomascft commented 2 years ago

Ok, so... I wrote a script with Github Actions, but I'm not sure how to implement the widget tests. Right now i just require("lain.widget"). Does that enumerate through the widgets below it or not. I could just make an array with all the widget's names but It'd be important to update that list whenever a new one is added.

thomascft commented 2 years ago

If you want to see what I've done so far, check the actions branch over here.

Aire-One commented 2 years ago

Hello,

That's nice to see some effort in this direction. The more the Awesome WM ecosystem (awesome itself and third-party modules / dependencies) has automatic test, the more it can be considered as mature and stable.

Regarding our tests at Awesome, we have a similar test suite for our integration tests than the one you propose here : https://github.com/awesomeWM/awesome/blob/master/tests/run.sh. It starts Xephyr and run step-by-step tests withing a graphical instance of Awesome. I think, the biggest downside of it, is that it's a homegrown tool, so not exactly what you expect from a conventional/standardized tool, and you have to learn it from scratch. It's not hard, but I'm not sure how easy it would be to reuse a similar script on another context, like for the lain module.

Unit testing is done with busted (https://olivinelabs.com/busted/). Because unit tests don't load a full working environment with Awesome, but only the part that is tested, we need to mock dependent modules in our tests, and to have this preload script (https://github.com/awesomeWM/awesome/blob/master/spec/preload.lua). It's a common annoyance of unit testing that you can't avoid. I'm also not sure how bad it would be for third-party modules.

(just my 2cts about CI testing Awesome WM)