exposure option selects worst items instead of best?!

[machow]

Hey, I've really been enjoying mirtCAT! One thing I've noticed recently is that the item exposure option doesn't seem to select from the {x} top items meeting a criteria. For example, if I use the first demo from the mirtCAT docs, with "MI" for criteria, and set the exposure to 2 for each item, then it selects an item that is clearly not one of the top 2.

I've included plots below from a run with exposure set to 2, and also the code I used to run it (it's basically the first example in the mirtCAT help doc). When I look at the item information at each stage of the test, it doesn't seem to match up with what was selected.

Code for adaptive test

set.seed(1234)
nitems <- 50
itemnames <- paste0('Item.', 1:nitems)
a <- matrix(rlnorm(nitems, .2, .3))
d <- matrix(rnorm(nitems))
dat <- simdata(a, d, 1000, itemtype = 'dich')
mod <- mirt(dat, 1)
coef(mod, simplify=TRUE)

# alternatively, define mo from population values (not run)
pars <- data.frame(a1=a, d=d)
mod2 <- generate.mirt_object(pars, itemtype='2PL')
coef(mod2, simplify=TRUE)

# simple math items
questions <- answers <- character(nitems)
choices <- matrix(NA, nitems, 5)
spacing <- floor(d - min(d)) + 1 #easier items have more variation in the options

for(i in 1:nitems){
    n1 <- sample(1:50, 1)
    n2 <- sample(51:100, 1)
    ans <- n1 + n2
    questions[i] <- paste0(n1, ' + ', n2, ' = ?')
    answers[i] <- as.character(ans)
    ch <- ans + sample(c(-5:-1, 1:5) * spacing[i,], 5)
    ch[sample(1:5, 1)] <- ans
    choices[i, ] <- as.character(ch)
}

df <- data.frame(Question=questions, Option=choices, 
                              Type = 'radio', stringsAsFactors = FALSE)
# include scoring by providing Answer key
df$Answer <- answers

pat <- generate_pattern(mod, Theta = 0, df = df)
res_MI <- mirtCAT(
  df, mod, 
  criteria = 'MI', 
  start_item = 'MI', 
  local_pattern = pat,
  design = list(exposure = rep(2, extract.mirt(mod, 'nitems')))
  )

plot(mod, type = 'infotrace')
plot(res_MI)

Examining selection

Here I sort item information calculated for each estimated theta during the test. If you filter on, say, information estimates at the 2nd point in the test, the the selected item (item 1) is pretty low on the list.

library(tidyverse)

thetas <- c(res_MI$thetas_history)

item_info <-
  extract.mirt(mod, 'itemnames') %>%
  tibble(item_num = 1:length(.), item_id = .) %>%
  mutate(
    item =  map(item_id, ~ extract.item(mod, .x)),
    info = map(item, ~ tibble(pos = 1:length(thetas), theta = thetas, info = iteminfo(.x, thetas)))
    ) %>%
  unnest(info) %>%
  arrange(pos, desc(info))

item_info %>% filter(pos == 2)      # item 1 is very low on list!

[philchalmers]

Thanks for the report and reproducible example! I definitely agree something is buggy here, particularly because the start_item input didn't even give the correct value (item 20 should have been selected first). I'll look into the cause of this issue.

[philchalmers]

Simple mistake on my part, the behaviour should now be correct on the dev version. Thanks again.

[machow]

Thanks for the quick fix!