elixir-lang / elixir

Elixir is a dynamic, functional language for building scalable and maintainable applications
https://elixir-lang.org/
Apache License 2.0
24.55k stars 3.38k forks source link

Very high memory consumption on big test suite due to context merged into ExUnit.Test structs #12141

Closed vorce closed 2 years ago

vorce commented 2 years ago

Elixir and Erlang/OTP versions

Erlang/OTP 25 [erts-13.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Elixir 1.13.4 (compiled with Erlang/OTP 25)

Operating system

Both macOS 12.5.1 and Debian (seen on both local execution as well as our CI)

Current behavior

Hello folks!

This week we’ve noticed an increase in our quite large test suite getting killed on CI servers (16GB RAM, 8vCPUs). The test suite consists of 18221 test cases, most of them async.

After some investigation we found that the ExUnit.TestModule structs can become very large (~137MB for a single phoenix controller test with ~110 test cases). Digging further we see that the test context is included in each ExUnit.Test struct is sent to the formatters.

Unfortunately for us we added our OpenApiSpex spec to the context in the Web.ConnCase module 🤦

We did this out of convenience, since the spec is needed to make assertions on if a response matches a spec (which we use a lot). The result is that the (very big!) spec is in every ExUnit.Test struct for our controller tests! We've fixed this, but still have some questions:

When we adjusted our test suite to remove the specs from the test context our memory usage dropped from ~9GB-16GB+ to a measly ~1.3GB. So, the gains from this are very significant.

As an aside, before that running without a formatter also already helped reduce memory usage.

If you need more details we’re happy to share! Thanks a lot.

Expected behavior

We’re unsure about the behavior of contexts being merged and kept as part of the ExUnit.Test struct - it seems to us that there could be room for improvement here (removed or reduced).

Alternatively, guidance/docs to restrict context usage would be great.

PragTob commented 2 years ago

And a thank you for all your excellent work! 💚

IMG_20220728_203124

josevalim commented 2 years ago

Awesome job @vorce! A fix will be available in the next Elixir version!

vorce commented 2 years ago

Wow that was fast. Thanks José! It was a big team effort at Remote 😃

PragTob commented 2 years ago

Thanks for the quick fix @josevalim !

IMG_20200510_081811_Bokeh