search

SeattleTestbed / utf

Unit Test Framework for SeattleTestbed
MIT License
1 stars 9 forks source link

Should unit tests be stateless? #66

Open aaaaalbert opened 9 years ago

aaaaalbert commented 9 years ago

We don't have a policy for whether unit tests must "clean up after themselves" or not, i.e. may leave traces of the state / fact of their exceution or not. Some tests clean up, some don't despite causing significant artifacts that changes the behavior of later re-runs through the same test.

Pros of requiring cleanup

A middle ground could be to somehow mark tests that require a re-build after they have run. Then at least it's clear that problems may arise.

Reaching out to @JustinCappos, @vladimir-v-diaz, @awwad in particular --- What do you think?

JustinCappos commented 9 years ago

I think you should remove state before running a test, not after. In other words, put the system state in the correct state and then run. If you clean up after a failure, then you may delete useful debugging information.

On Fri, Oct 2, 2015 at 7:23 AM, aaaaalbert notifications@github.com wrote:

We don't have a policy for whether unit tests must "clean up after themselves" or not, i.e. may leave traces of the state / fact of their exceution or not. Some tests clean up https://github.com/SeattleTestbed/seash/blob/master/tests/ut_seash_subprocess.py#L48-L67, some don't despite causing significant artifacts that changes the behavior of later re-runs through the same test https://github.com/SeattleTestbed/seash/blob/master/tests/ut_seash_allmodules.py#L21-L23 . Pros of requiring cleanup

  • If unit tests fail, this often means you will change code in response, and re-run the tests. Stateless tests speed this up a lot --- no re-cloning, re-building, etc.; Running the test again is like starting over anew.
  • Forces you to think harder before/while you implement. You will see problems like this in production code for sure, so why not train yourself on writing tests beforehand?

Cons of requiring cleanup

  • It's not always obvious that unit tests have side effects, and what these will be.
  • Cleaning things up correctly is complex, and probably requires lots of duplicate code across tests.

A middle ground could be to somehow mark tests that require a re-build

after they have run. Then at least it's clear that problems may arise.

Reaching out to @JustinCappos https://github.com/JustinCappos and @vladimir-v-diaz https://github.com/vladimir-v-diaz in particular --- What do you think?

— Reply to this email directly or view it on GitHub https://github.com/SeattleTestbed/utf/issues/66.

aaaaalbert commented 9 years ago

Sure, makes sense.

Any feelings on whether cleaning up is a MUST, SHOULD, or MAY in the sense of RFC 2119? :wink:

(Currently, the decision is left to the implementor.)

JustinCappos commented 9 years ago

In the case of success I think it would be a SHOULD.

However, there are also pre and post scripts for certain test modules (groups of tests) and the complexity of having per-test cleanups on success while leaving state on failure, is a mess.

The tests are here to make our lives easier. If being very rigid about our test behavior causes a lot of work, that is not helpful. However, needing to debug some specific test in a different way may cause us a lot of work over time. I would at a minimum expect any "outlier" tests to include copious comments explaining how they work and why.

(We should already have this for some of the resource tests that may crash the system if the repy VM does not handle them correctly.)

On Fri, Oct 2, 2015 at 12:17 PM, aaaaalbert notifications@github.com wrote:

Sure, makes sense.

Any feelings on whether cleaning up is a MUST, SHOULD, or MAY in the sense of RFC 2119? [image: :wink:]

(Currently, the decision is left to the implementor.)

— Reply to this email directly or view it on GitHub https://github.com/SeattleTestbed/utf/issues/66#issuecomment-145075199.