Hunt down non-deterministic cache test failures

HackerExperience / Helix

GNU Affero General Public License v3.0

53 stars 10 forks source link

Hunt down non-deterministic cache test failures #306

Closed renatomassaro closed 6 years ago

renatomassaro commented 6 years ago

A few remarks:

It happens a lot more often on slow systems (travis)
On faster systems, most of the intermittent failures happen on cold-cache tests.
Somehow bypasses Ecto.Sandbox (not sure if related)

renatomassaro commented 6 years ago

Insight: increasing the timeout of Channel-related tests (assert_reply, assert_push, assert_broadcast) reduces the amount of errors, specially those of "Account already created" and "Ecto ownership error".

It is probably unrelated to Cache, and it may account for the third bullet.

renatomassaro commented 6 years ago

Third bullet is solved on #322

Turns out two things happened:

Some tests were within the describe block, but weren't wrapped into the test/2 macro from ExUnit. This led them to completely ignore the Helix.Test.Case.Integration sandbox.
There was one or two tests that defined their setup file as module attributes. This led to a SQL query outside the Ecto sandbox.

renatomassaro commented 6 years ago

Bullets one and two were fixed on #329. So this issue is closed.

It happened because one test (queue_sync_test.exs) set the Sync interval to 50ms, which somehow impacted other tests, leading them to sync before it was supposed to.

For extra caution I've also modified the sync interval on the TEST environment to never be called.