cheezman34
commented
6 years ago Another option:
3) Keep a dict of {node_id : {pending_test1, pending_test2 ...}} and then just call len on the set. Remove and add items to the set as they're assigned/finished.
As each test is marked complete _pending_of is called at least once. _pending_of then iterates over a dictionary that contains a True/False for every assigned test. This has n^2 performance in the number of tests, which can really have an effect when you have a few thousand tests.
2 simple fixes come to mind: 1) Keep a dictionary of {node : pending_count} and increment/decrement as necessary. Then _pending_of is just a lookup instead of an iteration. 2) Delete finished nodeids and scopes from self.assigned_work to make the iteration faster.
Thoughts? Option 1 seems preferable to me, as then you're still keeping track of finished tests.