Add better model comparison diagnostics

athowes / multi-agyw

Spatio-temporal estimates of HIV risk group proportions for AGYW across 13 priority countries in sub-Saharan Africa

https://athowes.github.io/multi-agyw

MIT License

2 stars 2 forks source link

Add better model comparison diagnostics #24

Closed athowes closed 2 years ago

athowes commented 3 years ago

Perhaps R-INLA inbuilt CPO works OK here? Start by adding that
Want to be able to say what the diagnostic means, i.e. how good is the model in a more objective sense

athowes commented 3 years ago

[x] Make the best entry in each information criteria bolded
[ ] Add this for the single DHS as well

athowes commented 3 years ago

best_idx <- df %>%
  group_by(iso3) %>%
  summarise(
    dic_best_idx = list(which(dic == min(dic))),
    waic_best_idx = list(which(waic == min(waic))),
    cpo_best_idx = list(which(cpo == max(cpo))),
    pit_best_idx = list(which(pit == max(pit)))
  )

#' Adding bold for best value to the table
for(i in seq_along(min_ind)) {
  table <- table %>%
    tab_style(
      style = cell_text(weight = "bold"),
      locations = cells_body(columns = ?, rows = ?)
    )
}

athowes commented 3 years ago

Main thing left to do here is work out a way to have a (single) plot which shows all of the information criteria at once.

Average over countries?
Standardise by e.g. dividing by maximum DIC so that the average over countries isn't weighted towards bigger countries
Would prefer it not to have many facets for the sake of putting it in manuscript though that would be possible

athowes commented 2 years ago

Closing this. Point left to think about: what if the criteria disagree?