CI check that metrics defined in a custom Jetstream config are required to have associated statistical treatments

mozilla / metric-hub

Central hub for metric definitions that are considered the source of truth

https://mozilla.github.io/metric-hub

Mozilla Public License 2.0

14 stars 14 forks source link

CI check that metrics defined in a custom Jetstream config are required to have associated statistical treatments #353

Open danielkberry opened 9 months ago

danielkberry commented 9 months ago

Currently, a user can define a metric and add it to the list of experiment metrics, but if that metric is missing a statistical treatment it will be silently dropped and no analysis will be done on it. Users intend to do analysis with these metrics.

mikewilli commented 8 months ago

Hopefully this will be solved by the solution described in https://github.com/mozilla/metric-hub/issues/354