Hi, I ran the experiments for ood_locgpn{10,16} to reproduce the results. The metrics are logged after training as follows:
val
test
accuracy
0.923077
0.889306
brier_score
0.280527
0.269254
ECE
0.10638
0.0702577
confidence_aleatoric_apr
0.966722
0.980251
confidence_epistemic_apr
0.95776
0.962611
confidence_structure_apr
nan
nan
confidence_aleatoric_auroc
0.706276
0.864282
confidence_epistemic_auroc
0.635031
0.772384
confidence_structure_auroc
nan
nan
CE
0.37979
0.339846
avg_prediction_confidence_aleatoric
0.816697
0.819579
avg_prediction_confidence_epistemic
14919.8
14410.5
avg_sample_confidence_aleatoric
0.816697
0.819579
avg_sample_confidence_epistemic
16556.9
15927.3
avg_sample_confidence_features
15835.7
16779.6
avg_sample_confidence_neighborhood
nan
nan
average_entropy
0.513569
0.501767
ood_detection_aleatoric_apr
0.801985
0.78415
ood_detection_aleatoric_auroc
0.833289
0.836685
ood_detection_epistemic_apr
0.80048
0.788122
ood_detection_epistemic_auroc
0.852889
0.855747
ood_detection_features_apr
0.75896
0.748484
ood_detection_features_auroc
0.81794
0.8237
ood_detection_neighborhood_apr
nan
nan
ood_detection_neighborhood_auroc
nan
nan
ood_detection_structure_apr
nan
nan
ood_detection_structure_auroc
nan
nan
ood_accuracy
0
0
ood_avg_prediction_confidence_aleatoric
0.556919
0.558297
ood_avg_prediction_confidence_epistemic
2879.92
2808.5
ood_avg_sample_confidence_aleatoric
0.556919
0.558297
ood_avg_sample_confidence_epistemic
4068.24
4109.26
ood_avg_sample_confidence_neighborhood
nan
nan
ood_avg_sample_confidence_features
2063.81
1759.3
ood_average_entropy
1.03177
1.03767
id_accuracy
0.923077
0.889306
id_avg_prediction_confidence_aleatoric
0.816697
0.819579
id_avg_prediction_confidence_epistemic
14919.8
14410.5
id_avg_sample_confidence_aleatoric
0.816697
0.819579
id_avg_sample_confidence_epistemic
16556.9
15927.3
id_avg_sample_confidence_features
15835.7
16779.6
id_average_entropy
0.513569
0.501767
Is this experiment exclusive for the With Effect case? I would like to reproduce 3 cases: aleatoric w/Net, Epist w/Net and Epist w/o Net as table 1 in the paper. Could you provide me with some guidance? Thank you.
Hi, I ran the experiments for ood_locgpn{10,16} to reproduce the results. The metrics are logged after training as follows:
Is this experiment exclusive for the With Effect case? I would like to reproduce 3 cases: aleatoric w/Net, Epist w/Net and Epist w/o Net as table 1 in the paper. Could you provide me with some guidance? Thank you.