search

shanejansen / touchstone

Touchstone is a testing framework for your services that focuses on component, end-to-end, and exploratory testing.
16 stars 2 forks source link

Database Not Reset #13

Closed ScottFreeCode closed 3 years ago

ScottFreeCode commented 3 years ago

When running Touchstone Develop with snapshot_databases: true, after running an individal test, the database is no longer reset when running mocks reset or running all tests.

I'm not certain whether the reset stops happening or whether the snapshot gets overwritten (which would start getting reset to after the test state).

ScottFreeCode commented 3 years ago

For what it's worth I'm discovering this reproduces inconsistently. Sometimes using mocks reset fixes it, sometimes it doesn't.

As far as I've noticed it doesn't happen in the first place if I'm only running all tests, though, so it still seems to have something to do with running an individual test (or perhaps with doing something else after running an individual test).

(It would be nice if there were no need to run mocks reset ever because it would always reset before running a test, whether running all or individual, and merely would not reset after an individual test.)

shanejansen commented 3 years ago

@ScottFreeCode You shouldn't have to run mocks reset unless you are doing exploratory testing and want to revert mocks to their default state. Mocks should always get reset to their default state before a test is ran (all and individual)

shanejansen commented 3 years ago

@ScottFreeCode Oh boy! I spent way too much time on this before realizing it's doing exactly what it's supposed to! When running an individual test, Touchstone will not reset any mocks. This is a feature allowing the user to inspect mocks manually and track down bugs. Mocks are always reset before tests are ran so it will not affect tests in any way.

ScottFreeCode commented 3 years ago

Mocks are always reset before tests are ran so it will not affect tests in any way.

This is the part that's not happening. Once it gets into the bad state, running tests (either individually or all tests) doesn't run them against the initial state of the DB.

Is it possible the initial DB snapshot gets replaced by a snapshot of the DB after the individual test (or before running the subsequent test), such that it is resetting but is now resetting to a state from after the test? It should only save the DB snapshot at the end of services start (of course, it also should reset the DB before beginning services start so saving the state at the end cannot pick up any test state).

ScottFreeCode commented 3 years ago

For reproducing, I've noticed I seem to be able to avoid it if:

(I also haven't tried whether it still happens if I use the mysql defaults yaml instead of the service's migrations, i.e. set the snapshot option to false; since we got the snapshot option I've always preferred it.)

That suggests it might be reproducible by:

  1. Write a test "A" that alters the DB state and another "B" that checks the DB is in its initial state.
  2. Run test A individually.
  3. Do not call mocks reset.
  4. Run test B either individually or by running all tests.
shanejansen commented 3 years ago

Ah you're right!