Caboose APA - Get the tests passing

[reidlw]

Tests are failing; @AmeanAsad and @willscott are working on finding and fixing bugs.

[reidlw]

@AmeanAsad please leave notes, links to PRs/issues/etc. here to hand off for your PTO

[AmeanAsad]

Update and PTO Handoff

Relevant PR: https://github.com/filecoin-saturn/caboose/pull/153 Relevant Issue: https://github.com/filecoin-saturn/caboose/issues/150

Tried to provide a good baseline for tests that simulate the caboose pool refreshing over time. The main idea behind the tests is to establish a set of “good” nodes and a set of "bad" nodes. The tests simulate the caboose pool refreshing over time as these nodes are getting injected with stats. The pool should always converge to the "good" nodes set. I tried to keep things a bit verbose so that it is more readable (left some print statements, etc. that I was using for debugging) but expect to clean things up for the final version. The current tests have some high level description that clearly state what each test is achieving.

In terms of completion of the tests, the tests we currently have cover a few scenarios but here are the two biggest gaps I see we currently have in terms of testing the functionality:

• We are not testing how the pool adapts to a load factor on nodes. The current tests ignore how much load each individual node can take. I think we should have a test that establishes how the pool adapts to some nodes hitting their maximum load and starting to regress in performance.
• Testing how the adaptive algorithm impacts cache affinity. We should test the smoothness and rate of convergence of the pool between two sets of nodes. I tried to design the the tests such that it is a loop that refreshes the caboose pool. That way we can test the intermediate behaviour of the pool converging on some set of nodes.

[willscott]

Tests are now passing