search

weavejester / eftest

Fast and pretty Clojure test runner
424 stars 40 forks source link

Shorten test runs using LPT algorithmn #77

Open eraserhd opened 4 years ago

eraserhd commented 4 years ago

Since running test in parallel is exactly the "Multiprocessor scheduling problem", sorting the jobs by last known run time, then running longest first, should shorten the test time significantly (and also make run time more stable).

Proposal: We add a defonce atom with a map from test symbol to last known runtime in milliseconds. Sort the tests by this with some arbitrary default value if missing before assigning to threads, and record run time to the atom as tests complete. The first test run will effectively work the same way as it does today, but subsequent test runs should complete earlier. As the atom is just a heuristic, it is OK for it to be out of date.

I'm willing to do the work here, but does this sound good?

eraserhd commented 4 years ago

This'll require significant rework of the runner.

The process of running each namespace in a thread, and the namespace thread submitting each test to a fixed executor, should make the test order somewhat non-deterministic (even when the seed is fixed), but it also doesn't allow sorting through namespaces. So it would need one thread to order and one thread to reassemble the jobs and make NS end reports.

eraserhd commented 4 years ago

Some rough code to test the hypothesis shows about 8% improvement on a complicated test suite, which is nice, but not worth the investment for me at the moment.