Weird CI test failure in SSpec test infra

[noahgibbs]

Excerpt of the CI failure of the two tests:

TestSSpecInfrastructure
  test_skip                                                       PASS (10.71s)
  test_timeout_test_fail                                          FAIL (10.02s)
        Minitest result: Expected fail but got skip: Skipped, no message given!
        /Users/runner/work/scarpe/scarpe/test/test_helper.rb:90:in `run_scarpe_sspec'
        /Users/runner/work/scarpe/scarpe/test/test_helper.rb:40:in `block in run_scarpe_sspec_code'
        /Users/runner/work/scarpe/scarpe/scarpe-components/lib/scarpe/components/file_helpers.rb:23:in `with_tempfile'
        /Users/runner/work/scarpe/scarpe/test/test_helper.rb:39:in `run_scarpe_sspec_code'
        /Users/runner/work/scarpe/scarpe/test/test_sspec_infrastructure.rb:47:in `test_timeout_test_fail'

  test_many_assertions                                            PASS (10.98s)
  test_assertion_fail                                             PASS (14.[61](https://github.com/scarpe-team/scarpe/actions/runs/7766275781/job/21181907685#step:7:62)s)
  test_timeout_no_fail                                            FAIL (13.42s)
        Minitest result: Expected success but got fail: This should always fail!.
        Expected: true
          Actual: false!
        /Users/runner/work/scarpe/scarpe/test/test_helper.rb:90:in `run_scarpe_sspec'
        /Users/runner/work/scarpe/scarpe/test/test_helper.rb:40:in `block in run_scarpe_sspec_code'
        /Users/runner/work/scarpe/scarpe/scarpe-components/lib/scarpe/components/file_helpers.rb:23:in `with_tempfile'
        /Users/runner/work/scarpe/scarpe/test/test_helper.rb:39:in `run_scarpe_sspec_code'
        /Users/runner/work/scarpe/scarpe/test/test_sspec_infrastructure.rb:36:in `test_timeout_no_fail'

Not seeing this in PR test runs, but saw it from a merged-to-main test run.

[noahgibbs]

I should try this multiple times and see if it's timing based. Both are on timeout tests, which suggests it may be. Also all of these tests say they're taking 10+ seconds, which seems slow. Though maybe not -- GitHub Actions often runs tests very slowly with Webview.

[Nj221102]

@noahgibbs I tried recreating this issue both locally and by copying the repo and running tests on that copied repo but I was not able to do that, these timeout tests are passing both times, can you please tell me how to recreate these test fails?

[noahgibbs]

My guess would be that the timeout is too short for some CI workers. Sometimes GitHub Actions seems to have really slow/loaded hosts running the tests. I was seeing just over 10 seconds for the tests in that section. So maybe it just needs a longer timeout than 10 seconds? I assume that's the timeout since it's so close to the times in the excerpt.

The error doesn't happen on most CI runs, either. Time bugs are hard.

[noahgibbs]

For now I'm closing this. If we see it again, it's likely timeouts.