search

hyperfiddle / rcf

RCF – a REPL-first, async test macro for Clojure/Script
MIT License
278 stars 12 forks source link

rcf/set-timeout! incompatible with clojure.test which doesn't guarantee test order #82

Open didibus opened 10 months ago

didibus commented 10 months ago

I have some tests where I need to set a specific timeout, say to assert that something does timeout. But it seems when we run tests through CI, I can't control what timeout set for which tests.

Normally I do:

(do
  (rcf/set-timeout! 200)
  (tests "That it times out"
    ...))

But like I said, I feel this doesn't generate a deftest that will execute with a 200ms timeout always.

dustingetz commented 10 months ago

What platform, JVM? Can you demonstrate it? Please confirm this is under clojure.test integration only ({:jvm-opts ["-Dhyperfiddle.rcf.generate-tests=true"]}), cannot reproduce without that flag?

dustingetz commented 10 months ago

Can you reproduce using the flag locally outside of CI? What CI provider, github actions?

didibus commented 10 months ago

I'll try and see if I can find a minimal way to reproduce.

It's on JDK 17, OpenSuse tumbleweed Linux.

It's not actually on CI, just running clojure -X:test locally with cognitect test runner.

I did set the flag above, as well as the omit stack trace flag.

I believe the issue is that clojure.test doesn't guarantee the test order without an explicit test-ns-hook or something like that.

So the set-timeouts! are not executed prior to each test as they are defined in the test file. So if you have many tests, each one needing a different timeout, you can't control which test is executed with what timeout when running the tests with a test runner for clojure.test.