swaywm / sway

i3-compatible Wayland compositor
https://swaywm.org
MIT License
14.59k stars 1.11k forks source link

Create automated test to test the IPC commands #772

Closed deklov closed 8 years ago

deklov commented 8 years ago

I think this would be a very useful thing to have. Here is what I suggest:

deklov commented 8 years ago

@SirCmpwn your thoughts?

ddevault commented 8 years ago

I've already looked into unit tests (which is how I would prefer to do this sort of thing) and it didn't go well. At some point I may write a better test framework for C and then revisit the problem.

deklov commented 8 years ago

This would be more like integration or system tests, you typically want all three: unit, integration and system tests.

ddevault commented 8 years ago

I meant that we can test commands with unit tests instead of this sort of integration testing.

deklov commented 8 years ago

Okay, so lets start working on unit tests then.

deklov commented 8 years ago

What would be your preferred approach?

ddevault commented 8 years ago

Well first we'd have to find or write a unit testing framework for C that I like.

deklov commented 8 years ago

Okay. What unit testing frameworks to you like for other languages?

ddevault commented 8 years ago

I don't think good testing frameworks for C compare to good testing frameworks for other languages. Here's some basic requirements:

deklov commented 8 years ago

Perhaps it would be a good exercise to pick a few "units" in Sway and think about how we would test them if we had a perfect test framework with all the features you listed. What would be a good set of "units" for this?

deklov commented 8 years ago

I'm thinking this would help identify the exact features that we would want other than the one you listed and how we want them to be implemented.

ddevault commented 8 years ago

Here's the branch I abandoned last time I looked into tests. It has several tests in it.

deklov commented 8 years ago

The link wont load, it times out.

ddevault commented 8 years ago

Working on it.

ddevault commented 8 years ago

Should work now.

deklov commented 8 years ago

It does. Thanks.

deklov commented 8 years ago

That looks pretty cool. I saw that you have some nice tests for list and readline in there.

I've been looking at what needs to be unit tested to ensure the correctness of the move commands, its a lot, and some of it will not be easy to unit test. My experience is that it is sometimes quite hard to add unit tests after the fact. When production code and tests are written at the same time, the production code tends to be written in a way that makes it easy to test. If production is written without tests, the converse is often true.

Having given this some more thought, it seems to me that we can get a lot of value with minimal effort by doing system testing like I outlined in the description of this ticket. Unit testing would be preferred, and is a great long-term goal, but it just seems to be too much work and even with full unit test coverage these system test will still have value and the time spent implementing them will not have been wasted.

That's my two cents.

ddevault commented 8 years ago

I don't want to move forward with that strategy.

deklov commented 8 years ago

That's disappointing