emichael / dslabs

Distributed Systems Labs and Framework
https://ellismichael.com/dslabs/
1.23k stars 339 forks source link

Compute client waiting time without including testing framework time #50

Closed wilcoxjay closed 1 year ago

wilcoxjay commented 1 year ago

Previously, the time to stop node threads and the time to check invariants was charged against any outstanding client requests' waiting time. Stopping a node thread requires that the node thread finish executing the current event handler. If the event handler has a performance bug, that will cause runstate.stop() to be slow. Similarly, checking invariants can be expensive for long logs.

This commit introduces markEndTimeOfOutstandingRequest, which allows the test to set the ending time of outstanding requests before stopping threads and checking invariants. Then, after those (potentially expensive) operations, the tests can assert that the waiting time is acceptable.

Unfortunately this introduces a bit of a sharp corner into the testing framework's internal API: it would be easy to forget to call markEndTimeOfOutstandingRequest before assertMaxWaitTimeLessThan. Perhaps we should add a check for this to help with framework maintenance.

I also added a warning if the framework detects that runstate.stop() took longer than one second. That indicates that some event handler in the system took a long time to run. (This warning remains difficult to debug, but at least the new situation is better than the previous one.)

wilcoxjay commented 1 year ago

Fixes #48

emichael commented 1 year ago

Closing in favor of #54.