Write testsuite to file after each module is done

[vorce]

Hello @victorolinasc !

After #47 our test suite is using less memory. However even with that change (and https://github.com/elixir-lang/elixir/pull/11754 for the CLIFormatter), our big monolith's test suite with 12k+ test cases can hit the memory limits on (beefy!) CI runners. So this PR is my attempt at cutting out even more memory from JUnitFormatter. Instead of keeping things around until the end of the suite, this version will write each junit testsuite out to file as soon as a module is done. What do you think?

Thanks for a great project ❤️

[victorolinasc]

Hi @vorce !

I think this can be an option without a problem. I have a 8.1k unit tests project that runs fine (in 2m30s which is awesome) with less than 1.5GB RAM (which seems okay to me).

Just to be sure, you're saying your suite is using a LOT more than that right? I see no problems in adding this option at all, I just think that there might be something else going on too.

I'll take a look at the code now :)

[victorolinasc]

think I've misread the intention... sorry. The end result here would be the same but it just so happens that it will be more "stream" like than keeping all results.

Seems nice! I will try this out too on my suite and see how it goes!

Thanks once again!

[vorce]

Cool, yeah it should be fully compatible change 🤞 Let me know how it goes after trying.

Here's the summary of my measurements on our test suite with different formatter setups:

Baseline, CLIFormatter only

The CLIFormatter in elixir master with lower memory footprint.

measurement	memory bytes	memory GBs
max	3210920784	3.21
avg	1123334500	1.12

CLIFormatter + master JUnitFormatter

measurement	memory bytes	memory GBs
max	5880984816	5.88
avg	1821546739	1.82

CLIFormatter + this branch JUnitFormatter

measurement	memory bytes	memory GBs
max	4882953848	4.88
avg	1747206586	1.74

That even the baseline goes that high is a bit strange to me 🤔

[victorolinasc]

Not sure why but your branch did no difference em memory consumption for my case... it actually increased CPU usage but I can't trace that back exactly to this change... I'll try to debug it even more...

Just for curiosity: how are you measuring it?

[vorce]

Oh wow, weird.

I use this formatter together with the others to measure. It produces a csv file (that I then import into google sheets):

defmodule TestMem do
  use GenServer

  @impl true
  def init(_opts) do
    file = File.open!("memory_usage.csv", [:write, {:delayed_write, 1000, 200}])
    {:ok, %{file: file}}
  end

  @impl true
  def handle_cast({:test_finished, %ExUnit.Test{state: nil}}, config) do
    IO.binwrite(config.file, "#{DateTime.utc_now()},#{:erlang.memory(:total)}\n")
    {:noreply, config}
  end

  def handle_cast({:test_finished, %ExUnit.Test{state: {:failed, _failed}}}, config) do
    IO.binwrite(config.file, "#{DateTime.utc_now()},#{:erlang.memory(:total)}\n")
    {:noreply, config}
  end

  def handle_cast(_event, config), do: {:noreply, config}
end

[victorolinasc]

Nice idea :)

I'll try to dig deeper here (probably that the resource usage of the CI is not measuring properly) but it will take some more time. Sorry for that!

[vorce]

Yeah no problem! 🙌

[lukad]

I believe this can be closed. I recently added this formatter to a very large test suite (~40k tests) and didn't see any memory impact. Seems like the issue was fixed in ExUnit in https://github.com/elixir-lang/elixir/commit/7d4a58692ed70dd4d7cc44422fabdba73526567b and there should be no need for a workaround.

[vorce]

Yep I will close it! Seems it's not really needed anymore 👍