search

ooni / probe

OONI Probe network measurement tool for detecting internet censorship
https://ooni.org/install
BSD 3-Clause "New" or "Revised" License
750 stars 142 forks source link

webconnectivity LTE: A/B testing using richer input #2555

Open bassosimone opened 9 months ago

bassosimone commented 9 months ago

This issue is about performing A/B testing between webconnectivity v0.4 and LTE using richer input. We have already this functionality in tree since https://github.com/ooni/probe-cli/pull/1123. This issue is about using the check-in API to perform the actual A/B testing. We would collect data for either a selected subset of users or time. The goal would be to figure out the differences between these two versions, which then would inform which extra QA test to write and bugs to fix.

bassosimone commented 5 months ago

We're tentatively going to run this A/B testing around the next week for ooniprobe-android-unattended 3.8.6 only and take advantage of the staged rollout to observe changes. We're going to limit ourselves to such a specific version because it contains ooni/probe-cli@3.20.0, which ensures v0.4 and v0.5 generate the same results for our QA testing suite.