Implement AIC and BIC

[Leongrim]

The information criteria are used to compare results of different fit models. Using only the likelihood to compare the model with more free parameter will be preferred. That's why penalty terms are included in dependence of the number of free parameter.

BIC= k*ln(n)-2*ln(L)
https://en.wikipedia.org/wiki/Bayesian_information_criterion AIC= 2*k-2*ln(L)
https://en.wikipedia.org/wiki/Akaike_information_criterion k number of free parameter, n number of events, -ln(L) negativ log likelihood

The most tricky part will be to calculate the number of free parameter because this depended of the type of the variable. I do it like this in my script.

free_parameter=0
for parameter, value in fit_parameters.items():
    type(value)==np.complex: 
        free_parameter+=2
    elif type(value)==float: 
        free_parameter+=1
print(f"number of free parameter {free_parameter}")

[redeboer]

Ok agreed. The question is now where this is best implemented. That depends on how you as a user would like to use it.

Could you sketch some code snippet of what you would prefer the workflow to look like?

[Leongrim]

Like the code is currently written I think the best would be that the optimizer writes this into the fit result. As far as I can see, this is the only thing that knows which parameters are fitted(?) and also should have all the information to calculate the information criteria.

Something like this would be great:

minuit2 = Minuit2(
    callback=callbacks,
    use_analytic_gradient=False,
)
fit_result = minuit2.optimize(estimator, fit_parameters)
print(
    "AIC", fit_result.AIC,
    "BIC", fit_result.BIC,
    "#free parameter", fit_result.number_free_parameter,
    "#events", fit_result.number_fitted_entries,
    "nll", fit_result.estimator_value
)

[redeboer]

I start to doubt now whether it makes sense to add these measures. Reason is that AIC and BIC are just one of many criteria for selecting fit results and it is not the responsibility of TensorWaves to provide a selection of specific statistical techniques.

What could be provided is a way to count the number of degrees of freedom including the complex plane. That is more specific to optimization itself.

[Leongrim]

At least Tensorwaves should collect everything necessary in one place. So it should be sufficient if the degrees of freedom and the number of events fitted are included in the fit result. Than everything is in one place and the user doesn't has to collect it from many different objects. This is at least my opinion and I know that it is trivial to calculate the number of fitted events.