mozilla / self-repair-server

This project is now EOL, replaced by Normandy Recipe Server.
6 stars 11 forks source link

Sample more for the initial days of release #160

Open ghost opened 9 years ago

ghost commented 9 years ago

Upon new releases of Firefox, the HB score is too noisy due to throttling / updates being slow on initial rollout resulting in a smaller sample of users.

Proposed solution for heartbeat-by-user-first-impression for the Release channel (for now):

  1. Add a sampleMultiplier function that considers the version, channel, and current date to provide an additional multiplier to the sample rate. This will be utilized in shouldRun.
  2. sampleMultiplier will utilize a manually curated (for now) table to look up the release date for the version of the client. We will use date math to see how many days it has been since the release date. Let's call this diff daysSinceRelease for now
  3. We will use daysSinceRelease to calculate the sample multiplier based on historical rollout data. This may be a heuristic formula or a lookup table

Notes

Rob will make a first attempt on this. Gregg will review the code. Ilana and Gregg will review the heuristic.

MattGrimes commented 9 years ago

Solid. Do you have an ETA on when you might have the first attempt done?

ghost commented 9 years ago

I may be being overly optimistic, but I don't think that this is a difficult project. I'd say we could test for 42 without issue.

gregglind commented 9 years ago

Spec clarifications: