I think I found out why this regression was introduced. CI runs mix test.all but I didn't see any logs of the slow tests being run in CI. I tried running mix test.all locally and indeed they all pass because the slow tests don't run. I tried to narrow the problem down and found that if I have test.all defined as "test.all": ["test", "test --only slow"], then the slow tests don't run. But if I have "test.all": ["test --only slow"], then the slow tests do run and the expected failure shows up. I'm not exactly sure why the slow tests get skipped when there's a previous test that gets run.
From https://github.com/commanded/eventstore/issues/283#issuecomment-2016570951