The analysis is performed on top of the benchmarking data that records running times of each experiment. It was done too simplistically, because some methods call each other and may record double time in case they are run sequentially and not asynchronously.
Yet the goal is to calculate total system time spent for computations (and not wall time). Wall time would favor decentralized computation too heavily, because it is asynchonous, while centralized is synchronous. On the other hand, time benchmarks from centralized simulations should account for overlapping methods, which is not done now.
This negatively biases centralized simulations results..
kabirkbr
commented
6 years ago
The analysis is performed on top of the benchmarking data that records running times of each experiment. It was done too simplistically, because some methods call each other and may record double time in case they are run sequentially and not asynchronously.
Yet the goal is to calculate total system time spent for computations (and not wall time). Wall time would favor decentralized computation too heavily, because it is asynchonous, while centralized is synchronous. On the other hand, time benchmarks from centralized simulations should account for overlapping methods, which is not done now.
This negatively biases centralized simulations results..