mlswg / mls-implementations

Coordination of implementation and interop specific details
113 stars 14 forks source link

Make gRPC test runner less thorough to allow more actors #110

Closed mulmarta closed 1 year ago

mulmarta commented 1 year ago

Currently, the number of permutations grows exponentially with the number of actors. The new function creates a single random permutation of evenly spread clients. We can run the test many times to get many permutations.

mulmarta commented 1 year ago

I like this solution because it's very flexible. However, we will have the same problem later -- for larger groups we will have to use a script-generation script, which will assign clients... somehow? So I wonder if we need the additional flexibility here, or should we just implement some algorithm for choosing combinations here?

For example this one. We pick an arbitrary partition of the actors into 3 sets, say A={0,3,...}, B={1,4,...}, C={2,5,...}. Then we test on all assignments of clients to A,B and C. So in the first assignment 50001 gets all actors in A, 50001 gets all clients in B and 50003 gets all clients in C, etc. This way each actor is tested on all clients.

bifurcation commented 1 year ago

Friendly amendment: We structure actor values as <name><number>, e.g., alice123, where the name indicates the library and the number indicates a distinct instance of that library. Then the tool can generate all the assignments of library -> name and then create numbered actors from the selected libraries for a given run. This is conceptually / combinatorically the same as your A=... B=... C=..., just making it visible in the name.

bifurcation commented 1 year ago

Of course, I suspect there's a phase change at some point, below which we care about all the permutations and above which we're fine with random assignments. So maybe we should just add a switch on the script that says which of those it does?

bifurcation commented 1 year ago

After discussion, let's just have a switch between the "test all cases" and "test a random case". It can go at the top level, affecting all of the scenarios.

bifurcation commented 1 year ago

Closing in favor of #125