Why final result is considered the last stat received in the test ?

opentok / opentok-network-test

Sample app to test network connectivity and statistics (bps, packet-lost)

MIT License

112 stars 56 forks source link

Why final result is considered the last stat received in the test ? #15

Open guiracine opened 8 years ago

guiracine commented 8 years ago

I am trying to understand how the network stats test works under the hood. I understand that the test stabilizes as the time goes up but is it the correct thing to do to present the final result as the last stat received ? Shouldn't we do an average of all stat values received ?

We implemented the test in both iOS and js. In the javascript version we observed that that packet lost ratio could be > 0.0 in the test but the last stat received is == 0.0 so we consider that there is a good connection when this is not the case...Thats why we thought that we should compute the average of the stat values.

dylanjha commented 7 years ago

👍 wondering the same thing, did you try saving the stats and computing averages yourself?

dylanjha commented 7 years ago

I'm experimenting with saving all the report objects and calculating the mean bitsPerSecond and packetLossRatioPerSecond values.

The thing I'm wondering is this line in the "How does it work" section:

During the test period, the video quality will stabilize, based on the available network connection quality.

It might be useful to try:

plot the data points out over a time series and see how long it takes to stabilize
discard outliers before calculating the mean so that we ignore early sporadic data points

wobbals commented 7 years ago

I think we'll be taking another look at this project in the coming months. In the meantime, I did start a separate project that uses the same API to accumulate periodic reports and produce less granular result: https://github.com/wobbals/opentok-mos-estimator - still a very rough draft, but it could be useful to get your feedback if you're looking at improving how to process getstats data.

dylanjha commented 7 years ago

@wobbals this looks cool. I would love to help move opentok-mos-estimator forward. For the way it is set up now, what is a reasonable timeframe to run this test and get an estimate?

For our use case, we're looking at running a 20s or 30s test and making an assessment based on that. Would opentok-mos-estimator fit that use case?

wobbals commented 7 years ago

@dylanjha Probably some modifications might be wise for periodic sampling -- I'll shoot you an email and we can discuss.