jmbeach
commented
3 years ago I see what you're saying. What about changing the tests to be as follows:
| Producer Machines | Producer Instances | Consumer Machines | Consumer Instances | Message Size |
|---|---|---|---|---|
| 3 | 8 | 3 | 8 | medium |
| 3 | 8 | 3 | 8 | large |
| 3 | 8 | 3 | 8 | x-large |
| 4 | 25 | 4 | 25 | medium |
| 4 | 25 | 4 | 25 | large |
| 4 | 25 | 4 | 25 | x-large |
Does this address the issues and seem like a useful set of tests?
The experiment configuration for this makes no sense. We should be varying the pub/sub configuration for each message size. Theoretically you would have the first 5 iterations from test 1 and then multiply them for each iteration/variation in test 2 (5 iterations). This would result in 25 tests. You could lower this number by 5 because the first test has a message size of “small”. I personally don’t think we would need 25 tests… I would recommend running more of the extreme cases like tiny, medium, x large. And maybe even cut out some of the test 1 tests (i.e. just us the first 3 cases involving 1 machine per consumer/producer).