Issue with the estimated theta

[1sh1vam]

In 1 PL model I have all the items with high theta (>2). We start the assessment with theta 0. If the first item response is wrong new estimated theta is going above 2 and every subsequent false response is also increasing the theta. initializer = FixedPointInitializer(0) selector = MaxInfoSelector() estimator = NumericalSearchEstimator()

[douglasrizzo]

You mean you have all items with high b's (difficulties), but start the examinees' thetas in a place in the ability scale in which no items exist? Why would you want to do that?

Regardless, I'll try to replicate your scenario. Can you send me a minimal working example to speed it up? Just a script in which you initialize your item bank and run a simulation.

[1sh1vam]

Thanks Douglasrizzo for responding. We are using this library in an advance course where there are students who got enrolled without merit. So here is the data that we tested with.

This is an advance course there are students got into the course without merit. We are trying Here is an example itembank (1PL Model) items = np.array([[ 1. , 2 , 0. , 1. ], [ 1. , 2.31013589, 0. , 1. ], [ 1. , 2.61, 0. , 1. ], [ 1. , 2.91, 0. , 1. ], [ 1. , 3.1 , 0. , 1. ], [ 1. , 3.2, 0. , 1. ], [ 1. , 3.3, 0. , 1. ], [ 1. , 3.4, 0. , 1. ], [ 1. , 3.5, 0. , 1. ], [ 1. , 3.6, 0. , 1. ]])

Here is an script which simulates the problem mentioned above ` responses = [] administered_items = []

initializer = FixedPointInitializer(0) selector = MaxInfoSelector() estimator = NumericalSearchEstimator() est_theta = initializer.initialize() print('Examinee initial proficiency:', est_theta)

def getNextItem(resp = False): global items, selector, estimator, est_theta, responses, administered_items item_index = selector.select(items=items, administered_items=administered_items, est_theta=est_theta) print('Next item to be administered:', item_index)

administered_items.append(item_index) responses.append(resp) print('Current proficiency', est_theta) est_theta = estimator.estimate(items=items, administered_items=administered_items, response_vector=responses, est_theta=est_theta) print('New Estimated proficiency, given answered items:', est_theta)`

[1sh1vam]

Here is a link to colab notebook

[douglasrizzo]

Sorry for the delay in the response. I took a look at your Colab.

If I understand your particular situation, you have an item bank in which all items are too hard for examinees to answer any item correctly.

In cases where the examinee's response vector is composed entirely of wrong or right answers, the maximum likelihood estimator estimates their proficiency as $-\infty$ and $\infty$, respectively. By default, catsim does not do that and employs a different estimation method in these special cases, which is documented here: https://douglasrizzo.com.br/catsim/cat.html#catsim.cat.dodd

Dodd's method is used to compute a "placeholder proficiency" to guarantee numerical stability to the CAT estimation procedure while the examinee does not break the all-wrongs or all-rights pattern. But, in your case, that pattern persists until the end of the test.

Just note that, even though the examinee's proficiency is getting larger, it is always under the difficulty of the easiest item in the bank, which means Dodd's method still places the examinee outside your proficiency scale. The fact that the proficiency is getting larger should be irrelevant, as all your student's that get all items wrong will still be outside your instrument's evaluative capabilities.

If you don't want to use Dodd's method, you just have to create the estimator like so: NumericalEstimator(dodd=False). You should then get $-\infty$ as a proficiency estimation for your examinees.

You could also choose a different initial $\hat\theta$ for the examinee's, e.g. the mean of the difficulty parameter in your item bank.

[1sh1vam]

Thanks Douglasrizzo for your response. I think you are correct even if proficiency is increasing it's still less than the easiest item proficiency, Which makes sense.

[douglasrizzo]

Glad it worked for you!