Should the add-on expire by itself after 7 weeks (instead of the current arbitrarily set 20 weeks)?

[motin]

@mlopatka

From https://github.com/motin/taar-experiment-v3-shield-study/issues/39#issuecomment-403235034:

Note however that until mozilla/shield-studies-addon-utils#232 is resolved, we can only expect cleanup to run upon expiration of the add-on, and not when the user/normandy uninstalls/disables the add-on.

One workaround for this is to avoid Normandy uninstallation as the main means of ending the study, and instead set the expiration time to exactly 7 weeks (as per the PDH study duration), after which the study ends itself. The current expiration is currently set to 20 weeks in order to act as a safety measure to ensure expiration in case normandy does not successfully uninstall the add-on.

In essence, we would start instead relying on the study self-expiration to be the main method of ending the study, and instead use normandy unenrollment as a safety measure instead of the main unenrollment method.

Note: QA expects the study to expire after 7 weeks.

[mlopatka]

In essence, we would start instead relying on the study self-expiration to be the main method of ending the study, and instead use normandy unenrollment as a safety measure instead of the main unenrollment method.

This is a pretty substantial change from the typical SHIELD process. Historically, we have relied on Normandy for both SHIELD study enrollment targeting.

@gregglind I believe you and @motin have had preliminary discussions on this topic, but I want to make sure that the implications of adding a hard-coded termination date at as a study specific are communicated through to QA and those responsible for Normandy targeting.

The main process implications that I can think of are:

1. Reliance on Normandy for all targeting task (no targeting sanity check in the study addon)
2. Reliance on the study addon itself for a clean uninstall (Normandy acts only as an emergency failsafe to remove misbehaving addons)
3. QA / Firefox engineering responsible for ensuring that the uninstallation code is harmonized with the PHD design, as the Normandy unenrollment is usually initiated by relman based on the PHD (I believe).
4. Deciding on a good threshold between study-enforced enrollment date and Normandy cleanup will need to be based on the rolling enrollment of clients into each study such that for a study-enforced unenrollment of x weeks, the last enrolled client gets x weeks in the study and is not unenrolled by Normandy before the study addons has a chance.
5. b) perhaps this needs to be reflected in the PHD template?

Where else do we imagine this change affecting workflows, stakeholders? I'm hoping that implementing the study uninstall at the study addon level allows us to have cleaner uninstalls (in terms of resetting prefs that the study affects), but I want to make sure that we are not creating blind spots elsewhere in the SHIELD process.

Please feel free to flag anyone else who who should be part of this discussion.

@biancadanforth @pdehaan @rhelmer @mythmon

[gregglind]

+1 on wanting a reliable, consistent way to:

• ensure studies run long enough
• ensure studies stop running after a timestamp, or after they run long enough

Expiration is a thorny topic, as you are pointing out. All solutions here has failure modes. It's unclear, which failure modes we care about most.

1. It would be nice if we had canonical 'the time is this' in Firefox / experiments. Not having this means that all 'client-side' expiration is based on client clocks. We currently can't even guarantee those are monotonically increasing
2. When addons expire, it all looks the same. This is potentiall fixable in Firefox, by adding "reason" codes for uninstalls, so that Normandy, User, Expiration, and other types of uninstall are distinguishable.

Speaking to the specific proposal that all studies have expiration times, I think the experiments program is agnostic. I claim it is up the analyst to decide that. I support making it a real field in the QA / PhD plan, as part of a general "all the ways this can undeploy".

So my official answer is to waffle and wait to see if we can fix it for real, upstream.

GL

[motin]

Thanks @gregglind. Seems that we'll go with 7 weeks expiration for this specific study and continue the discussions in the utils repo issues on how we find a reliable, consistent way to do this in the future.