NorskRegnesentral / shapr

Explaining the output of machine learning models with more accurately estimated Shapley values
https://norskregnesentral.github.io/shapr/
Other
147 stars 34 forks source link

Workaround for failing test on macOS-latest on GHA #407

Closed martinju closed 1 month ago

martinju commented 1 month ago

For some unknown reason, the output_categorical_asym_causal_mixed_cat test fails on macOS-latest, where the first explicand gets a single differnet sample v(S={2}). I have verified that the sample on GHA are indeed valid. The same is the ones we get on all other OS and locally. Changing seed or nor increasing the number of mc_samples does not fix the issue, but simply dropping the 3 explicand gives indentical result (even though the first is the one with the different results). We'll apply this workaround for now as we cannot find a good reason for the differences. Also adding dt_samp_for_vS = TRUE to more easily spot the differences if reverting later. The below output sample shows the differences in row 2:

> new$internal$output$dt_samp_for_vS[id_coalition==3 & id==1]
   id_coalition    id Month_factor Ozone_sub30_factor Solar.R_factor Wind_factor     w p_hat1
          <int> <int>       <fctr>             <fctr>         <fctr>      <fctr> <num>  <num>
1:            3     1            7                  0      (269,301]           6   0.4 82.562
2:            3     1            8                  0      (203,236]           8   0.2 84.690
3:            3     1            7                  0      (269,301]           5   0.2 84.388
4:            3     1            7                  0      (170,203]           5   0.2 80.465
> old$internal$output$dt_samp_for_vS[id_coalition==3 & id==1]
   id_coalition    id Month_factor Ozone_sub30_factor Solar.R_factor Wind_factor     w p_hat1
          <int> <int>       <fctr>             <fctr>         <fctr>      <fctr> <num>  <num>
1:            3     1            7                  0      (269,301]           6   0.4 82.562
2:            3     1            7                  0      (236,269]           7   0.2 58.197
3:            3     1            7                  0      (269,301]           5   0.2 84.388
4:            3     1            7                  0      (170,203]           5   0.2 80.465