Closed mulmarta closed 1 year ago
I like this solution because it's very flexible. However, we will have the same problem later -- for larger groups we will have to use a script-generation script, which will assign clients... somehow? So I wonder if we need the additional flexibility here, or should we just implement some algorithm for choosing combinations here?
For example this one. We pick an arbitrary partition of the actors into 3 sets, say A={0,3,...}, B={1,4,...}, C={2,5,...}
. Then we test on all assignments of clients to A,B
and C
. So in the first assignment 50001
gets all actors in A
, 50001
gets all clients in B
and 50003
gets all clients in C
, etc. This way each actor is tested on all clients.
Friendly amendment: We structure actor
values as <name><number>
, e.g., alice123
, where the name indicates the library and the number indicates a distinct instance of that library. Then the tool can generate all the assignments of library -> name
and then create numbered actors from the selected libraries for a given run. This is conceptually / combinatorically the same as your A=... B=... C=...
, just making it visible in the name.
Of course, I suspect there's a phase change at some point, below which we care about all the permutations and above which we're fine with random assignments. So maybe we should just add a switch on the script that says which of those it does?
After discussion, let's just have a switch between the "test all cases" and "test a random case". It can go at the top level, affecting all of the scenarios.
Closing in favor of #125
Currently, the number of permutations grows exponentially with the number of actors. The new function creates a single random permutation of evenly spread clients. We can run the test many times to get many permutations.