Module is no longer copied when using Elixir 1.17 RC with --repeat-until-failure option

[mbuhot]

While trying out the Elixir 1.17 release candidate I noticed that when using the new --repeat-until-failure option, the tests run successfully the first time they are executed, but then on the second run the Mimic copies appear to be lost?

Console output (partially redacted)

➜ ✗ mix test test/my_app/some/test.exs --repeat-until-failure 100   
Running ExUnit with seed: 891520, max_cases: 20

........
Finished in 1.9 seconds (1.9s async, 0.00s sync)
8 tests, 0 failures
Running ExUnit with seed: 829513, max_cases: 20

  1) test Some test (MyApp.Some.Test)
     test test/my_app/some/test.exs:364
     ** (ArgumentError) Module DateTime has not been copied.  See docs for Mimic.copy/1
     code: fixtures([
     stacktrace:
       (mimic 1.7.4) lib/mimic.ex:498: Mimic.validate_server_response/2
       (my_app 0.1.0) test/support/util.ex:39: MyApp.Test.Util.stub_utc_now/1
       test/my_app/some_test.exs:96: MyApp.Some.Test.fixtures/1
       test/my_app/some_test.exs:372: (test)

The DateTime module is copied in the test_helper.exs:

Mimic.copy(DateTime)

The stub_utc_now util function in the stack trace is roughly:

  def stub_utc_now(now) do
    Mimic.stub(DateTime, :utc_now, fn ->
      now |> DateTime.shift_zone!("Etc/UTC") 
    end)
  end

[edgurgel]

Interesting 🤔 I will check it out! thanks!

[harrisi]

I think the issue is that --repeat-until-failure [n] doesn't re-run test/test_helper.exs. I'm not sure if it really makes sense for Elixir to do that, but maybe it's worth opening an issue or PR.

[harrisi]

Not an issue with Elixir. Mimic.copy/1 is calling Mimic.Server.reset/1 in a function passed to ExUnit.after_suite/1, here.

I think it's safe to just remove that line.

EDIT: I just saw the fix/repeat-until-failure branch. How embarrassing. Anything holding that back?

[edgurgel]

@harrisi, you are right that we should be able to essentially avoid the reset if we have repeat-until-failure happening. I don't know why but I'm getting flaky specs on that branch so I parked for a bit! I intend to come back to it soon but if anyone wants to try to understand why and fix it please feel free to!