search

growthbook / growthbook

Open Source Feature Flagging and A/B Testing Platform
https://www.growthbook.io
Other
5.8k stars 469 forks source link

[Bug] Risk thresholds for metrics configurable fr frequentist stats engine #1214

Open bosyk opened 1 year ago

bosyk commented 1 year ago

Summary

When using the frequentist stats engine, I can change bayesian properties for a metric.

Expected Behavior

When using the frequentist stats engine, I can only change behaviour relevant for frequentist statistics. An example would be the section from v1.9.0

image

Current Behavior

When using the frequentist stats engine, I can change behaviour relevant for bayesian statistics. An example would be the section from v2.0.0

image
lukesonnet commented 1 year ago

Thanks for opening this issue. The reason this is currently still an option is that we envision the possibility of being able to see results in both engines within the same experience and letting you toggle between the two on the fly. Allowing this would mean we should also allow people to configure settings that apply to one engine or the other.

That said, we should figure out a way to improve the UX so that users who only want one engine or the other don't see settings that don't apply to them.

As such, I'm not sure I would describe this as a bug, but I agree the UX is sub-optimal.

bosyk commented 1 year ago

I agree, it's not a bug, but an unexpected ed change in the UI's behaviour. I'm looking forward to the features and improvements you teasered 😉