search

ctmm-initiative / ctmmweb

Web app for analyzing animal tracking data, built upon ctmm R package
http://biology.umd.edu/movement.html
GNU General Public License v3.0
30 stars 21 forks source link

Alternative information criteria in model selection #107

Closed chfleming closed 3 years ago

chfleming commented 4 years ago

Occasionally, such as with small samples, there is need to use an information criterion (IC) other than the default approximate AICc. It would be useful to have a dropdown box somewhere before model selection to choose an IC other than approximate AICc. This argument would then be fed into the IC argument of ctmm.select and its model list summary. Current ICs supported are "AICc", "AIC", "BIC", "LOOCV", and "HSCV".

SandraCuadros123 commented 4 years ago

Hi Christen. Thanks, this would be very useful. I had an issue with some of my data because pf the small sample sizes probably and this would really help to solve the problem.

xhdong-umd commented 4 years ago

@chfleming When you used a different IC, summary over multiple attempts models still says AICc? Should that be updated to the actual IC used?

If this is the case, this could be a little bit difficult as the model summary code rely on the column name, and I need to make it work for all possible values...

res_1 <- ctmm.select(tele_list[[1]], tele_guess_list[[1]][[2]], IC = "BIC", verbose = TRUE)
> summary(res_1)
                     ΔAICc ΔRMSPE (m) DOF[area]
OU anisotropic    0.000000  575.84409  26.43320
OUF anisotropic   2.226091  567.44310  26.68302
OUf anisotropic  13.275649   22.30848  50.22548
OU isotropic     24.572981  282.07571  29.28846
OUF isotropic    26.515583  261.64516  30.58978
IID anisotropic 150.133919    0.00000  99.00000
chfleming commented 4 years ago

You have to provide the IC argument to both ctmm.select and summary. Then it should change.

xhdong-umd commented 4 years ago

Got it. Though this means all my model summary code need to be able to handle all possible IC column names, that will not be easy. I'll check how it can be done.

xhdong-umd commented 3 years ago

LOOCV seem to take extra time to run ctmm.select.

chfleming commented 3 years ago

ICs that require extra calculation like LOOCV do also require being set in ctmm.select. LOOCV is also an O(n^2) algorithm, so it is very slow, but it's only necessary for tiny datasets.

chfleming commented 3 years ago

Also, even if the IC is calculated in the fit object, if it wasn't set in ctmm.select then you can ensure that the appropriate models were attempted.

xhdong-umd commented 3 years ago

IC option is added. This is like adding a knob to a pipe line of machines, a lot of places need to be adjusted, and it has been so long since I worked on this part so it took extra time to pick up all the complex details.