new_theta from estimator

[naokishibuya]

Hello Mr. Douglas,

I have a question about the theta estimation returned from the estimator in the example Colab notebook.

The example code goes like below:

# response data
responses = [True, True, False, False]
administered_items = [1435, 3221, 17, 881]

# create objects
initializer = FixedPointInitializer(0)
selector = MaxInfoSelector()
estimator = NumericalSearchEstimator()
stopper = MinErrorStopper(.2)

# initialize the theta
est_theta = initializer.initialize()

# new theta estimation
new_theta = estimator.estimate(items=items, administered_items=administered_items, response_vector=responses, est_theta=est_theta)

# stop?
_stop = stopper.stop(administered_items=items[administered_items], theta=est_theta)
item_index = selector.select(items=items, administered_items=administered_items, est_theta=est_theta)

# simulate a response
true_theta = 0.8
a, b, c, d = items[item_index]
prob = icc(true_theta, a, b, c, d)
correct = prob > random.uniform(0, 1)

administered_items.append(item_index)
responses.append(correct)

Then, I should go back to the new theta estimation step.

My question is whether I should replace est_theta with new_theta before re-calling the estimator method. But then, it would replace the est_theta with the result based on only a few responses, which is probably not good. But when conducting adaptive testing for a new user (say, est_theta is initialized to 0), this approach may make sense (or not).

An alternative (as in the example) is to keep using the est_theta as initialized and accumulating responses until the stop condition is met. In this case, the final estimate is the new_theta, which is not used in the process. In this case, the stop condition and item selection are based on the initial theta. Therefore, the information is based on the initial theta, too.

So, Is either approach appropriate based on the usage? Or I should always follow the way the example indicates. I'd like to know your opinion.

[douglasrizzo]

Hi Naoki, thanks for showing an interest in catsim. You could ignore new_theta and assign the new estimation directly to est_theta, like I do below.

I put everything inside a loop so that the simulation only stops when the stopper object says so.

# initialize empty lists to keep response data
responses = []
administered_items = []

# create objects
initializer = FixedPointInitializer(0)
selector = MaxInfoSelector()
estimator = NumericalSearchEstimator()
stopper = MinErrorStopper(.2)

# initialize theta
est_theta = initializer.initialize()

# true theta value for simulation purposes
true_theta = 0.8

while not stopper.stop(administered_items=items[administered_items], theta=est_theta):
    # select item to present the examinee with
    item_index = selector.select(items=items, administered_items=administered_items, est_theta=est_theta)

    # simulate a response
    a, b, c, d = items[item_index]
    prob = icc(true_theta, a, b, c, d)
    correct = prob > random.uniform(0, 1)

    # store test data
    administered_items.append(item_index)
    responses.append(correct)

    # estimate new theta
    est_theta = estimator.estimate(items=items, administered_items=administered_items, response_vector=responses, est_theta=est_theta)

[naokishibuya]

Hi Douglas,

It makes perfect sense now.

Thanks!

[naokishibuya]

Thanks, I'm closing the issue.