ValueError: Only binary classification is supported. The type of the target is multiclass.

[nchen909]

Hi Markus, Thanks for sharing your excellent work! When I run Calibration - (Temp, Iso, Beta, Hist).ipynb df_iso = cal_results(IsotonicRegression, PATH, files, {'y_min':0, 'y_max':1}, approach = "single") I got resnet110_c10

---

ValueError Traceback (most recent call last) /tmp/ipykernel_443943/2066161569.py in ----> 1 df_iso = cal_results(IsotonicRegression, PATH, files, {'y_min':0, 'y_max':1}, approach = "single")

~/autodl-tmp/NN_calibration/scripts/calibration/cal_methods.py in cal_results(fn, path, files, m_kwargs, approach) 291 probs_val[idx_nan] = 0 292 --> 293 # Get results for test set 294 error, ece, mce, loss, brier = evaluate(softmax(logits_test), y_test, verbose=True, normalize=False) 295 error2, ece2, mce2, loss2, brier2 = evaluate(probs_test, y_test, verbose=False, normalize=True)

~/autodl-tmp/NN_calibration/scripts/calibration/cal_methods.py in evaluate(probs, y_true, verbose, normalize, bins) 201 202 y_prob_true = np.array([probs[i, idx] for i, idx in enumerate(y_true)]) # Probability of positive class --> 203 brier = brier_score_loss(y_true=y_true, y_prob=y_prob_true) # Brier Score (MSE) 204 205 if verbose:

~/miniconda3/lib/python3.8/site-packages/sklearn/metrics/_classification.py in brier_score_loss(y_true, y_prob, sample_weight, pos_label) 2690 y_type = type_of_target(y_true, input_name="y_true") 2691 if y_type != "binary": -> 2692 raise ValueError( 2693 "Only binary classification is supported. The type of the target " 2694 f"is {y_type}."

ValueError: Only binary classification is supported. The type of the target is multiclass.

How can I solve it? Thanks! Best, nchen909

[nchen909]

conda create -n calibration python=3.7 pip install scikit-learn==0.20.1 pip install betacal solve the problem.