Percentage changed without a reason

[ViseLuca]

Hi, We have a project that since yesterday in the CI/CD pipeline, and in local env, changed the coverage percentage.

mix coveralls from 89,2% is 88.9% mix coveralls.html is remaining 89,2% with all the tests

In my local env is changing if I run it 10 times 7 are correct, 3 are not

There is a way to discover why is changing sometimes?

[PaulOstazeski]

Are you running elixir 17 on erlang 27 by any chance? I've started seeing some unexpected changes to excoveralls output after upgrading erlang and elixir.

[PaulOstazeski]

So... I'm wrong about the elixir/erlang versions being our culprit. It turns out that we have some hidden dependencies within our test suite such that running mix test --cover twice without changing anything, the code does take different paths and correctly reports different coverage. For example, a test that hits this function will have "flaky" coverage reporting. (ours aren't this, but for the sake of example)

def is_this_covered do
  if :rand.uniform(2) == 1 do
    covered_half_the_time()
  else
    covered_the_other_half_of_the_time()
  end
end