Closed AlexanderFengler closed 1 month ago
This is indeed a corner case - this check is only performed when there are two choices in the data (len(np.unique(data["response"])) == 2
), which indicate that even though there are supposed more than two choices, only two unique values were found in the data.
This example shows that inferring the number of choices from data is unreliable. In multiple-choice cases certain choices can be legitimately missing. The most robust solution to this is to require the users to provide the number of choices (but also leave a default at 2 to be compatible with existing code), and we can warn the users of potential missing choices in their data and raise errors when illegal choices show up
In the context of running models that allow choice options of
[0, 1, 2, 3]
, I ran into an error due to_pre_check_data_sanity()
which reads as:This is likely a corner case since it does not always happen.