beyond-all-reason / teiserver

Middleware server for online gaming
https://www.beyondallreason.info/
MIT License
50 stars 47 forks source link

Fix inconsistent "mix test" results based on order of execution #366

Closed DeviousNull closed 1 month ago

DeviousNull commented 1 month ago

Currently, several tests will pass/fail depending on whether they run before or after other tests. This happens because some tests create lobbies but do not remove them, while other tests assume that there are no lobbies when they begin.

For all tests which currently create lobbies, this PR:

I have used the following command to test this PR:

Using the current master branch without these patches, this command results in:

Finished in 21.0 seconds (4.9s async, 16.0s sync)
441 tests, 19 failures, 162 excluded

Randomized with seed 468657

With these patches, the above command results in:

Finished in 70.0 seconds (5.0s async, 65.0s sync)
441 tests, 0 failures, 162 excluded

Randomized with seed 468657
DeviousNull commented 1 month ago

@geekingfrog : Hmm. I have a suspicion that there's a performance-related factor in why the tests fail for me...

For Teiserver development/testing, I use a VM with an iSCSI-connected disk. Having your database process going out over the network in order to access the actual data-on-disk is, let's say, slightly slow.

It is extremely consistent to reproduce when I use that seed, though, so this PR does appear to improve the robustness of the tests at a minimum.

jauggy commented 1 month ago

On Master

1

I get 1 error on master and on this PR.

Teiserver.Logging.Tasks.PersistServerDayTaskTest [test/teiserver/logging/persist_server_day_task_test.exs]
  * test perform task (31.1ms) [L#7]

  1) test perform task (Teiserver.Logging.Tasks.PersistServerDayTaskTest)
     test/teiserver/logging/persist_server_day_task_test.exs:7
     ** (KeyError) key :date not found in: nil. If you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map
     code: assert log.date == Timex.to_date({2021, 1, 1})
     stacktrace:
       test/teiserver/logging/persist_server_day_task_test.exs:17: (test)

On PR

3

Same error as above. Weird.

However this PR does make the tests more robust so it's probably good overall.