app crashes if uploading data that includes 3 or more previous tests

[rbcavanaugh]

the issue right now is almost certainly related to not having a potential starting item to pick from because these four in "choices" have been exhausted previously.

get_first_item <- function(previous_dat = NULL){

  choices = c(130, 25, 39, 154)

  if(is.null(previous_dat)){

    # if we want to sample the four anyway
      # first_item = sample(choices, 1)
      # return(first_item)
    return(130)

  } else {

    first_item <- 
      tibble(choices = choices) %>%
      filter(choices %!in% previous_dat$slide_num) %>%
      slice_sample(n = 1) %>%
      pull(choices)

    print(first_item)
    return(first_item)
}
}

These four are available because they're all the closest to 0 in item difficulty. Probably we should pick another 4-5 and include a second statement that says if length(first_item) == 0, then choose from the other set.

[gfergadiotis]

I would like to point out that up until now, we have only studied the behavior of the CAT algorithm for two administrations. I suspect that more than that, and certainly 4+ administrations, using only the PNT items, might be problematic. The reason is that the first couple of administrations (and especially if the second is a variable length CAT) would exhaust the majority of informative items, and therefore the 3rd+ administrations will have too much error. Also, they will most likely be super frustrating for patients, because patients will be presented with off-target items (a bunch of too easy or too hard items). All this to say, that I don't think we should spend time on allowing the ShinyApp to administer more than 3 CATs. And, even that might be too much and should come with a disclaimer that as items are exhausted, repeated administrations beyond the second one might yield ability estimates with too much error.

This will become more relevant when when add more items to the item bank. If you have the bandwidth to do it now, I would suggest a somewhat different approach:

For the first administration, always pick the first item. For any subsequent administration, use the final ability estimate, θ, and identify the item whose difficulty parameter is closest to the final θ. Α recent thesis in my lab suggested that there is no real advantage in terms of estimation efficiency. However, this approach does not require specifying a priori a list of items.

[rbcavanaugh]

All this to say, that I don't think we should spend time on allowing the ShinyApp to administer more than 3 CATs.

Perfect, that resolves the bug. @AlexSwiderski we will need to add this to the documentation and I think throw a warning/error when more than 2 previous administrations are uploaded.

For the first administration, always pick the first item. For any subsequent administration, use the final ability estimate, θ, and identify the item whose difficulty parameter is closest to the final θ. Α recent thesis in my lab suggested that there is no real advantage in terms of estimation efficiency. However, this approach does not require specifying a priori a list of items.

Oh - perfect and this would reduce the code base. I am assuming that if 2 previous administration are ok, then we should use the most recent one to determine the starting point?

This sounds good to me. I'll let alex/will weight in too.

[william-hula]

I agree with the comments and decisions up to now. I suspect that even a 3rd CAT that excludes previously administered items could be pretty low quality for a lot of people, and I think the documentation should have an explicit warning to that effect. Also, would it be possible to permit clinician to choose whether or not to exclude previously administered items? Doing that is a nice feature, but it's probably not always necessary, e.g, if you have 6 months or more between assessments.

[rbcavanaugh]

• ok it is now optional to exclude items from the last testing administration
• There is no option to exclude all previously administered items (i.e. for more than 1 previous test) given these concerns.
• first item in second test is selected using the final ability estimate from the previous administration

I still need to re-run the test to make sure this hasn't broken anything but everything looks good. I will re-run the test after fixing a few more minor issues.