Tests executing out-of-order with `lein test`

[ptaoussanis]

Hi Jay,

Thanks for all your work on the lib!

Any chance you could explain (or point me to a resource explaining) what determines the order in which tests are executed? I was working under the assumption that they execute on a single thread, linearly top-to-bottom - but I'm running into results that seem to disagree.

When I got to debugging and adding print statements to each test body, I'm getting the following out of lein test:

"5 6 7 8 1 2 3 4" (i.e. the last 4 tests are executing before the first 4).

Don't have a lot of time to dig into this atm, so any pointers to get me started would be greatly appreciated. Cheers! :-)

Environment: Expectations 1.4.56, Clojure 1.5.1, Lein 2.3.3

[jaycfields]

Hi Peter, how are you running your tests? lein-expectations? expectations-mode? in IntelliJ?

[ptaoussanis]

Hi Jay, am using lein-expectations 0.0.8 (so lein expectation, not lein test - sorry).

Will try get a minimum reproducible example to you in a couple days. But basic principle here: tests are supposed to execute linearly on one thread, yes?

Thanks again!

[jaycfields]

One thread - definitely

I haven't focused on ensuring that they run in any specific order, but I can take a look at what it would take to ensure ordering.

[ptaoussanis]

Okay, great- thanks Jay. Juggling a few urgent things atm - will try follow up once I know if the problem's on my end or not.

Cheers! :-)

[jaycfields]

Hopefully I emailed on this before I closed it, but I'm adding this now, in case I didn't.

I put printlns in a bunch of expected values and they printed in order. I ran the tests using expectations-mode, at the repl, and using lein expectations. All 3 produced the output in order.

So, I don't know of any reason they'd run out of order.

[ptaoussanis]

Hi Jay,

Have a reproducible example of tests apparently running out-of-order in some public code.

Using Expectations 2.0.12, lein expectations "taoensso.carmine.tests.message-queue" at the command line produces output:

→9
9→10
10→1
1→2
2→3
3→4
4→5
5→6
6→7
7→8

Ran 10 tests containing 10 assertions in 1324 msecs
0 failures, 0 errors.

So test 9 is running first, then 10, then 1.

This is currently in the https://github.com/ptaoussanis/carmine.git repo, dev branch if you want to inspect it locally.

I've been unsuccessful in trying to pin down the necessary conditions for a more minimal reproducible case. But even if we assume my code's buggy - it appears that (?) the tests themselves may be running out-of-order?

Any input would be hugely appreciated. I'll update you if I find anything further that might help diagnostically.

[jaycfields]

Thanks, that's really helpful. I was able to reproduce the same thing locally. I found what I believe is the issue in expectations, fixed it, and released 2.0.13 this morning. I also tested it with carmine and saw correct ordering.

Thanks for helping me get this resolved.

Cheers, Jay

[ptaoussanis]

Jay, all tests appear to be passing (incl. command-line tests) with v2.0.13. You are a champion, thank you so much for getting this resolved! I owe you one.

I've taken a look at the fix and I'm curious: any idea under what circumstances would the sort-by actually be necessary? Why/when would the vars be coming in in anything other than line order?

[jaycfields]

You can pass vars to that function, so a custom runner may pass them in out of order. The standard path would enter that function from

https://github.com/jaycfields/expectations/blob/master/src/clojure/expectations.clj#L313

where the vars are created here

https://github.com/jaycfields/expectations/blob/master/src/clojure/expectations.clj#L309

which is fairly straightforward: https://github.com/jaycfields/expectations/blob/master/src/clojure/expectations.clj#L298-L303

I supposed the sort-by str works in the majority of situations, and is the reason it was tough to track down a case where things didn't work as expected.

[jaycfields]

ns-interns returns a map, which I wouldn't expect to be sorted, so I'll always have to sort somewhere... I assume.

[ptaoussanis]

I supposed the sort-by str works in the majority of situations, and is the reason it was tough to track down a case where things didn't work as expected.

No kidding. I was seeing the weirdest things: like the tests would pass from the command line iff I imported an empty namespace, or included an empty macro, or rearranged some fn definitions that weren't even involved in the tests.

I'm happy to see the cause+solution ended up being so simple. Ahh, computers :-)

All the best, cheers! :-)