bassosimone
commented
8 months ago I've split this large issue into three issues for miniooni, mobile, and ooniprobe:
- https://github.com/ooni/probe/issues/2603
- https://github.com/ooni/probe/issues/2604
- https://github.com/ooni/probe/issues/2605
The original issue was a good placeholder but also too large. These three issues should be actionable.
This issue is about refactoring how we run experiments such that we always and unconditionally call check-in before any experiment (or group of experiments). In turn, this means that we have updated feature flags (which were introduced as part of richer input). In turn, this means that the information on whether to run experimental experiments is always fresh.
This issue is a follow-up of https://github.com/ooni/probe/issues/2553.
As a reminder, experimental experiments are experiments that we just added, so we're not 100% sure about them, or experiments that are historically flaky, so we want to be defensive.
We should refactor quite soon because News Media Scan could be refactored soon not to use check-in, therefore the quick hack described in https://github.com/ooni/probe/issues/2553 may soon become real technical debt.
Note: check-in flags expire after 24h, therefore the maximum window after which enabled-by-check-in experiments would be disabled again is 24h if the user does not perform any check-in in the meanwhile. That said, we still have potential future issues with News Media Scan, so better not to dally here 😅.