The big ol' compedium of flaky tests

SquidDev commented 1 year ago

CC:T's test suite is incredibly flaky when run via CI. I think this is just a limitation of how powerful the runners are (they work fine on my machine!) - but having to rerun every third build gets old quickly.

This issue is an attempt to track recent tests failures.

Gametests fail due to timeout

Builds

Build 2243: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4413979109/jobs/7735111084
Build 2257: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4430925396/jobs/7773273120, https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4430925396/jobs/7773650001 (only some failed here)
Build 2271: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4525631854/jobs/7970262900

Notes

In 2243, the test batch was created at 09:21:22 and finally started running at 09:21:25 (this is 3s for a single tick, which is Bad(TM)). Computers continue to turn on until 09:21:28, and then tests time out at 09:21:29 (after 150 ticks). There's two odd things here:

Computers really shouldn't take 3 seconds to all turn on - we're using the fake Kotlin machine here, which has very little startup overhead.
Tests time out after 150 ticks, which should take 7.5s, but only appears to be 4s. Unless Minecraft is skipping ticks here or something - maybe we should add some logging here just for debugging?

`modem_test.have_peripherals.modem_test.have_peripherals`

Starts with peripherals ==> 
 expected: <[monitor_0, printer_0, right]> 
  but was: <[right]> at 4,-58,22 (relative: 4,2,2) (t=1)

Builds

`inventory_test.fails_on_full.inventory_test.fails_on_full`

No peripheral: minecraft:furnace_0 at 47,-58,11 (relative: 2,2,1) (t=1)

Builds

Build 2268: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4515339240/jobs/7952515237

`modem_test.full_modems_form_networks.modem_test.full_modems_form_networks`

On the same network ==> expected: <dan200.computercraft.impl.network.wired.WiredNetworkImpl@554b3d66> but was: <dan200.computercraft.impl.network.wired.WiredNetworkImpl@c4809f9>

Builds

Build 2268: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4515339240/jobs/7952515237

`turtle_test.peripheral_change.turtle_test.peripheral_change failed`

expected: <[(peripheral_detach, right), (peripheral, right)]> 
 but was: <[(peripheral_detach, right)]>

Builds

Build 2270: https://github.com/cc-tweaked/CC-Tweaked/actions/runs/4525430871/jobs/7969917473

a2937 commented 20 hours ago

Pardon me, but where does the tests live and how do you run it?

SquidDev commented 16 hours ago

The game tests live in the projects/common/src/testMod folder and can be run with ./gradlew runGametest.

The contributing and architecture docs have a little more info.

cc-tweaked / CC-Tweaked