egouldo / ManyEcoEvo

Software for analysing Many-Analysts' style data and generating the ManyEcoEvo project data
https://egouldo.github.io/ManyEcoEvo/
GNU General Public License v3.0
2 stars 0 forks source link

Check random effects structure in deviation on peer-ratings models #153

Open egouldo opened 4 days ago

egouldo commented 4 days ago

See: https://github.com/egouldo/ManyAnalysts/issues/80#issue-2504983330

After re-running the systematic checks on the updated pipeline, the following holds:

  1. continuous ratings, $Z_r$: Reviewer ID as random effect
  2. categorical ratings, $Z_r$: effect ID / study ID as random effect
  3. continuous ratings, $y_i$: Reviewer ID is the only random effect generating converging, singular models
  4. categorical ratings, $y_i$: effect ID / study ID is the random effect that generates the most number of converging, non-singular models across all out-of-sample estimates for both datasets.

I note that the current version of the functions does not match these results:

https://github.com/egouldo/ManyEcoEvo/blob/580ce6d173b6d374249b468c4d43b99be47769a5/R/fit_boxcox_ratings_cat.R#L78-L79

https://github.com/egouldo/ManyEcoEvo/blob/580ce6d173b6d374249b468c4d43b99be47769a5/R/fit_boxcox_ratings_cont.R#L50-L51

egouldo commented 4 days ago

Local .Rprofile detected at /Users/elliotgould/Documents/GitHub/ManyEcoEvo/.Rprofile

library(tidyverse)
library(insight)
#> Warning: package 'insight' was built under R version 4.4.1
library(targets)

tar_read(ManyEcoEvo_viz) %>%
  filter(str_detect(model_name, "box_cox"), !str_detect(model_name, "no_int")) %>%
  filter(if_all(ends_with("_subset"), ~ .x == "All"), exclusion_set == "complete") %>%
  ungroup() %>%
  select(-ends_with("set"), dataset, starts_with("model_")) %>%
  unite(model_name, dataset, model_name, remove = F, sep = "-") %>%
  rowwise() %>%
  mutate(mod_call = list(insight::get_call(model))) %>%
  pull(mod_call, model_name)
#> Registered S3 method overwritten by 'parsnip':
#>   method          from 
#>   print.nullmodel vegan
#> Registered S3 method overwritten by 'lava':
#>   method         from    
#>   print.estimate EnvStats
#> $`blue tit-box_cox_rating_cont`
#> lme4::lmer(formula = box_cox_abs_deviation_score_estimate ~ RateAnalysis + 
#>     (1 | ReviewerId), data = data_tbl)
#> 
#> $`blue tit-box_cox_rating_cat`
#> lme4::lmer(formula = box_cox_abs_deviation_score_estimate ~ PublishableAsIs + 
#>     (1 | study_id), data = data_tbl)
#> 
#> $`eucalyptus-box_cox_rating_cont`
#> lme4::lmer(formula = box_cox_abs_deviation_score_estimate ~ RateAnalysis + 
#>     (1 | ReviewerId), data = data_tbl)
#> 
#> $`eucalyptus-box_cox_rating_cat`
#> lme4::lmer(formula = box_cox_abs_deviation_score_estimate ~ PublishableAsIs + 
#>     (1 | study_id), data = data_tbl)

Created on 2024-09-14 with reprex v2.1.0