search

mozilla / metric-hub

Central hub for metric definitions that are considered the source of truth
https://mozilla.github.io/metric-hub
Mozilla Public License 2.0
13 stars 13 forks source link

[ci skip-rerun] re-enable covariate adjustment #509

Closed danielkberry closed 3 months ago

danielkberry commented 3 months ago

Reverts mozilla/metric-hub#498.

When we previously pushed this, we encountered some errors. Those were largely fixed in https://github.com/mozilla/mozanalysis/pull/266. However, when an experiment is rerun, there is now a race condition between the analysis periods. Later analysis periods (e.g., week & overall) will depend on prior analysis periods (i.e., preenrollment_week) which may or may not have completed. In the case where covariate adjustment using preenrollment_week was specified but preenrollment_week is not available, we will now fall back (https://github.com/mozilla/mozanalysis/pull/268) to unadjusted inferences and emit a warning. Since these improvements are now in Jetstream (https://github.com/mozilla/jetstream/pull/2085), we can attempt to reenable the inferences.

Once the Jetstream rearchitecture is complete, we expect to be able to specify dependencies between periods and avoid the need to fallback.

github-actions[bot] commented 3 months ago

⚠️ Changing default metrics will cause all experiments that are currently live to get rerun after this change has been merged. This may come with a substantial cost. To skip reruns, add [ci skip-rerun] to the PR message.