tensorflow / similarity

TensorFlow Similarity is a python package focused on making similarity learning quick and easy.
Apache License 2.0
1.01k stars 104 forks source link

Calibration results at requested thresholds are not shown for some classification metrics (i.e. precision) #227

Closed MarinaZhang closed 2 years ago

MarinaZhang commented 2 years ago

There is an issue with using calibration on some metrics such as precision, where the results at the requested thresholds are not shown (only the 'optimal' result is being shown).

This is the expected result when using calibration with f1 (working).

calibration = sim_model.calibrate(
    x=x,
    y=y,
    calibration_metric='f1',
    extra_metrics=['precision', 'recall'],
    thresholds_targets={f"{v/10}": v/10 for v in range(10)},
)

Screen Shot 2022-01-26 at 3 46 37 PM

The bug when using calibration with precision - only the optimal result is shown, even though higher precision is requested in calibrate:

calibration = sim_model.calibrate(
    x=x,
    y=y,
    calibration_metric='precision',
    extra_metrics=['f1', 'recall'],
    thresholds_targets={f"{v/10}": v/10 for v in range(10)},
)

Screen Shot 2022-01-26 at 3 46 48 PM

owenvallis commented 2 years ago

Fixed in #267