search

tompave / fun_with_flags

Feature Flags/Toggles for Elixir
https://hexdocs.pm/fun_with_flags/FunWithFlags.html
MIT License
1.08k stars 79 forks source link

Suggest disabling cache for tests #171

Closed saveman71 closed 8 months ago

saveman71 commented 8 months ago

Nothing on the Testing section of the README.md suggests that cache may want to be disabled, but when running a lot of tests in parallel, the ETS cache may end up containing entries that are conflicting with other tests.

Should we recommend adding 

config :fun_with_flags, :cache,
  enabled: false

to config_test.exs?

tompave commented 8 months ago

Hey there, thank you for using the library.

Nothing on the Testing section of the README.md suggests that

If you mean this Testing section, then it's about running the tests for the library itself, and not running tests in a user project that is using the library.

I would think that users would know whether caching their flag data can interfere with their tests. I wouldn't explicitly recommend it because it could be confusing. If a user's tests are meant to traverse the stack, then it's a good idea to keep the set up similar to production. If in production their FWF uses the cache, then their tests should too.

saveman71 commented 8 months ago

Thank you for your answer!

Understood about the section, yes that's the one I was talking about but wasn't entirely sure it was for testing the lib or not.

The issue we had is that we are creating many users, with ids left up to ex_machina. What we've seen is this behavior (solved by disabling cache, or running the incriminated test with async: false:

image

It would return true, then 5s later return false.

So the theory is that the actor is cleared by another test and as the ETS table is global, the new call returns false even if it's in the same test.

This was only appearing very rarely only in tests, but in real life, ids wouldn't be shared and prod benefits from safe caching for sure.

I'll close the issue for now, as I believe its existence is already enough to be googleable ;)