During a recent run of backfill.py, I noticed the "No network connection" screen appearing instead of our test page. I noticed the test results for this 8/20 nightly build (3321ms median, 4384ms max) were higher than the other builds I ran at the same time (8/19: 3183ms median, 3328ms max). Upon re-running, the numbers dropped:
first re-run for 8/20 nightly: 3241ms median, 3915ms max
second re-run for 8/20 nightly: 3248ms median, 3295ms max
i.e. the median increased by 73ms under poor network conditions for some of the runs. To keep the results reproducible, we want to make sure we're not running the tests under no/poor network connectivity. Ideas:
before each run, use adb shell to test network connectivity (could affect time)
host the server on the local network to reduce network issues
During a recent run of backfill.py, I noticed the "No network connection" screen appearing instead of our test page. I noticed the test results for this 8/20 nightly build (3321ms median, 4384ms max) were higher than the other builds I ran at the same time (8/19: 3183ms median, 3328ms max). Upon re-running, the numbers dropped:
i.e. the median increased by 73ms under poor network conditions for some of the runs. To keep the results reproducible, we want to make sure we're not running the tests under no/poor network connectivity. Ideas:
Here's the raw data from my runs: test-results-network.zip