Create automated test to test the IPC commands

[deklov]

I think this would be a very useful thing to have. Here is what I suggest:

• Enable Sway to run in a headless mode
• Add a new command to dump a json representation of the layout tree and possibly other things
• Write a little script library with helpers to launch Sway, sent it commands and parse the results
• Use the above to write test cases for all commands
• Enjoy less manual testing and fewer bugs

[deklov]

@SirCmpwn your thoughts?

[ddevault]

I've already looked into unit tests (which is how I would prefer to do this sort of thing) and it didn't go well. At some point I may write a better test framework for C and then revisit the problem.

[deklov]

This would be more like integration or system tests, you typically want all three: unit, integration and system tests.

[ddevault]

I meant that we can test commands with unit tests instead of this sort of integration testing.

[deklov]

Okay, so lets start working on unit tests then.

[deklov]

What would be your preferred approach?

[ddevault]

Well first we'd have to find or write a unit testing framework for C that I like.

[deklov]

Okay. What unit testing frameworks to you like for other languages?

[ddevault]

I don't think good testing frameworks for C compare to good testing frameworks for other languages. Here's some basic requirements:

• Flexible support for mocking, likely via -Wl,--wrap=
• A nice set of assertions
• Running tests in a seperate address space so we can catch segfaults and sigaborts and such without blowing the whole test suite
• Written in C with at least moderately good code that we can understand+extend
• Not super buggy (which rules out cmocka, which I invested heavily in before because it met the first three criteria)

[deklov]

Perhaps it would be a good exercise to pick a few "units" in Sway and think about how we would test them if we had a perfect test framework with all the features you listed. What would be a good set of "units" for this?

[deklov]

I'm thinking this would help identify the exact features that we would want other than the one you listed and how we want them to be implemented.

[ddevault]

Here's the branch I abandoned last time I looked into tests. It has several tests in it.

[deklov]

The link wont load, it times out.

[ddevault]

Working on it.

[ddevault]

Should work now.

[deklov]

It does. Thanks.

[deklov]

That looks pretty cool. I saw that you have some nice tests for list and readline in there.

I've been looking at what needs to be unit tested to ensure the correctness of the move commands, its a lot, and some of it will not be easy to unit test. My experience is that it is sometimes quite hard to add unit tests after the fact. When production code and tests are written at the same time, the production code tends to be written in a way that makes it easy to test. If production is written without tests, the converse is often true.

Having given this some more thought, it seems to me that we can get a lot of value with minimal effort by doing system testing like I outlined in the description of this ticket. Unit testing would be preferred, and is a great long-term goal, but it just seems to be too much work and even with full unit test coverage these system test will still have value and the time spent implementing them will not have been wasted.

That's my two cents.

[ddevault]

I don't want to move forward with that strategy.

[deklov]

That's disappointing