iampelle
commented
3 months ago Hi,
”Information ratio” is basically just a fancy way to say “current sample size divided by final expected sample size”. So it would seem that, in your data, your sample size decreases one day compared to the previous. Could that be the case?
Cheers, Pelle (who unfortunately doesn’t maintain confidence anymore since I left Spotify in Dec. -23)
tis 16 juli 2024 kl. 23:29 skrev Eric Scott @.***>:
I'm trying to use the lib for the first time, for a sequential test. I'm getting an exception "ValueError: Information ratio must be monotonically increasing", but I don't know what it means. I've tried digesting the code, but this is buried more levels deep than I understand. Can you educate me?
Here's the snipped exception:
File ~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py:135, in compute_sequential_adjusted_alpha.
.adjusted_alphas_for_group(grp) 133 def adjusted_alphas_for_group(grp: DataFrame) -> Series: 134 return ( --> 135 sequential_bounds( 136 t=grp["sample_size_proportions"].values, 137 alpha=grp[ALPHA].values[0] / n_comparisons, 138 sides=2 if (grp[PREFERENCE_TEST] == TWO_SIDED).all() else 1, 139 ) 140 .df.set_index(grp.index) 141 .assign( 142 *{ 143 ADJUSTED_ALPHA: lambda df: df.apply( 144 lambda row: 2 (1 - st.norm.cdf(row["zb"])) 145 if (grp[PREFERENCE_TEST] == TWO_SIDED).all() 146 else 1 - st.norm.cdf(row["zb"]), 147 axis=1, 148 ) 149 } 150 ) 151 )[["zb", ADJUSTED_ALPHA]] File ~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py:53, in sequential_bounds(t, alpha, sides, state) 52 def sequential_bounds(t: np.array, alpha: float, sides: int, state: DataFrame = None): ---> 53 return bounds(t, alpha, rho=2, ztrun=8, sides=sides, max_nints=1000, state=state)
File ~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py:343, in bounds(t, alpha, rho, ztrun, sides, state, max_nints) 341 raise ValueError(f"Information ratio must must not be zero, {get_input_str()}") 342 if any(t[i] > t[i + 1] for i in range(len(t) - 1)): --> 343 raise ValueError(f"Information ratio must be monotonically increasing, {get_input_str()}") 344 if not (sides == 1 or sides == 2): 345 raise ValueError(f"sides must either be one a zero, {get_input_str()}")
File ~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py:337, in bounds.
.get_input_str() 334 def get_input_str(): 335 return ( 336 f"input params: t={t}, alpha={alpha}, sides={sides}, rho={rho}, ztrun={ztrun}," --> 337 f"state_df={state.df.to_json()}, state_fcab={state.last_fcab}, max_nints={max_nints}" 338 ) AttributeError: 'NoneType' object has no attribute 'df'
And here's the setup I did, trying to follow the example notebook:
test = conf.ZTest( metrics_df, numerator_column='metric1', numerator_sum_squares_column='metric1_sumsq', denominator_column='user_cnt', categorical_group_columns=['bucket'], ordinal_group_column='date' )
This call goes boom!
result = test.multiple_difference( level='control', groupby=['date'], level_as_reference=True, final_expected_sample_size_column='expected_users', non_inferiority_margins = (None, 'increase'), absolute=False )
— Reply to this email directly, view it on GitHub https://github.com/spotify/confidence/issues/102, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHJDXQWCMFVUCI6GFNENNDZMWGE7AVCNFSM6AAAAABK7MIQDCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYTEMBXGQ2TINY . You are receiving this because you are subscribed to this thread.Message ID: @.***>
scottee
I'm trying to use the lib for the first time, for a sequential test. I'm getting an exception "ValueError: Information ratio must be monotonically increasing", but I don't know what it means. I've tried digesting the code, but this is buried more levels deep than I understand. Can you educate me?
Here's the snipped exception:
And here's the setup I did, trying to follow the example notebook: