Problematic details in calculating quantities for evaluating DNN performance.

[elarichterwas]

17.08.2019

--> calculate_metrics from anal_py/anal_utils.py not corectly calculates mean of calc_pred_argmaxs_distances because is taking absolute value! change definition so the abs() is taken latter when needed, eg. for calculating fractions in a given bin.

--> original weights wt are not normalised to probabilities. Int. (A + B cos(alpha) + C sin(alpha) ) = 2 pi A predicted weights by DNN are normalised to probablities. Check that weights stored in files softmax_calc_w.npy softmax_preds_w.npy are normalised to unity integral before storing, so it does not have to be corrected at the analysis level.

Check that normalised weights are used for calculating
loss function.

Same problem in case of option with regression

16.08.2019

--> problem with reproducing oracle predictions by multi-class classifiation code. AUC=0.728 and not 0.782 for scalar-pseudoscalar case also present in case of nc=3, which should be equivalent results from binary classification.

could be tested when with scripts:
anal_py/anal_roc_auc_rhorho_nc3.py
anal_py/anal_roc_auc_rhorho_nc20.py

Likely, it comes from not correctly extended definitions
anal_py/anal_utils.py
   def test_roc_auc()
   def calculate_roc_auc()
   def evaluate_roc_auc()

and their original versions in src_py/tf_model.py

Note, that with original weights there is no problem and note the difference with the implementation used for calculating AUC see in scripts: anal_py/ anal_oracle_rhorho.py

[JanczarKurek]

As it is said we did not normalize preds and calc so it could be the reason roc_auc was not calculated properly.