Failing test when developing locally

[NoahTheDuke]

Hey friends, when I run the kaocha.watch-test/watch-test, I get the failure below. This happens when I git clone ...kaocha.git && cd kaocha && bin/kaocha, so before I've made any changes. Given that the tests pass in CI, I'm not sure what's wrong.

• OSX 12.6.6 on a 2019 Macbook Pro
• javac 11.0.20
• Clojure CLI version 1.11.1.1386

FAIL in kaocha.watch-test/watch-test (main.java:40)
expected: (str/includes? (clojure.core/deref out-str) (str/replace (str/replace "[(F)]\n\nFAIL in foo.bar-test/xxx-test (bar_test.clj:1)\nExpected:\n  :xxx\nActual:\n  -:xxx +:yyy\n1 tests, 1 assertions, 1 failures.\n\nbin/kaocha --config-file PATH --focus 'foo.bar-test/xxx-test'\n\n[watch] Reloading #{foo.bar-test}\n[watch] Re-running failed tests #{:foo.bar-test/xxx-test}\n[(F)]\n\nFAIL in foo.bar-test/xxx-test (bar_test.clj:1)\nExpected:\n  :xxx\nActual:\n  -:xxx +:zzz" "foo" prefix) "PATH" (str config-file)))
  actual: (not (str/includes? "[(F)]\n\nFAIL in foo15039.bar-test/xxx-test (bar_test.clj:1)\nExpected:\n  :xxx\nActual:\n  -:xxx +:zzz\n1 tests, 1 assertions, 1 failures.\n\nbin/kaocha --config-file /var/folders/5w/z50rbwg546x94k8p8jzqfbl00000gn/T/kaocha_integration6368814963772694138/tests.edn --focus 'foo15039.bar-test/xxx-test'\n\n[watch] Reloading #{foo15039.bar-test}\n[watch] Re-running failed tests #{:foo15039.bar-test/xxx-test}\n[(F)]\n\nFAIL in foo15039.bar-test/xxx-test (bar_test.clj:1)\nExpected:\n  :xxx\nActual:\n  -:xxx +:zzz\n1 tests, 1 assertions, 1 failures.\n\nbin/kaocha --config-file /var/folders/5w/z50rbwg546x94k8p8jzqfbl00000gn/T/kaocha_integration6368814963772694138/tests.edn --focus 'foo15039.bar-test/xxx-test'\n\n" "[(F)]\n\nFAIL in foo15039.bar-test/xxx-test (bar_test.clj:1)\nExpected:\n  :xxx\nActual:\n  -:xxx +:yyy\n1 tests, 1 assertions, 1 failures.\n\nbin/kaocha --config-file /var/folders/5w/z50rbwg546x94k8p8jzqfbl00000gn/T/kaocha_integration6368814963772694138/tests.edn --focus 'foo15039.bar-test/xxx-test'\n\n[watch] Reloading #{foo15039.bar-test}\n[watch] Re-running failed tests #{:foo15039.bar-test/xxx-test}\n[(F)]\n\nFAIL in foo15039.bar-test/xxx-test (bar_test.clj:1)\nExpected:\n  :xxx\nActual:\n  -:xxx +:zzz"))
╭───── Test output ───────────────────────────────────────────────────────
│ SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
│ SLF4J: Defaulting to no-operation (NOP) logger implementation
│ SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
╰─────────────────────────────────────────────────────────────────────────
1 tests, 1 assertions, 1 failures.

[alysbrooks]

Hi @NoahTheDuke, this test is one of the flakier ones in our suite, so it's not too surprising, unfortunately.

Here's a cleaner diff of actual versus expected:

I think there's a timing issue that causes the test to only be run after the file is changed and then run again, rather than being run once before and once after, which is what we'd expect and the behavior we're trying to test. It is being run twice, so I presume the actual functionality is still solid, and this test failure doesn't represent a real bug.

[lambduhh]

Attempted to reproduce on OSX 14.1.1 on a Apple M1 Pro by running Kaocha in watch mode on git@github.com:lambdaisland/deep-diff2.git First to verify watch mode was running smoothly I'd make small change to a file in tests upon saving change I'd verify Kaocha ran and reported the error. Then, I'd undo the change and save again to see if I could get Kaocha to replicate the behavior reported. For each version of java I ran this experiment ~5 times.

First tried to reproduce on my usual setup:

javac javac 18.0.1.1
Clojure CLI version version "1.11.1.1435"

Then noticed that the version of Java reported by OP was 11 so downloaded older version and attempted to replicate with same java version.

javac javac 11.0.21
Clojure CLI version version "1.11.1.1435"

Started to go down the rabbit hole of replicating clojure versions but since that requires uninstalling / reinstalling using a custom home-brew...with such a small diff decided not worth the effort without further evidence. Was not able to successfully replicate.