matthewcornell
commented
1 year ago The validation program _validate_quantile_prediction_dict() uses the helper function _le_with_tolerance() to compare a zipped offset of the incoming values:
def _validate_quantile_prediction_dict(prediction_dict, target):
...
is_le_values = [_le_with_tolerance(a, b) for a, b in zip(pred_data_values, pred_data_values[1:])]
if not all(is_le_values):
raise RuntimeError(f"Entries in `value` must be non-decreasing as quantiles increase. "
f"value column={pred_data_values}, is_le_values={is_le_values}, "
f"prediction_dict={prediction_dict}")def _le_with_tolerance(a, b): # a <= b ?
if type(a) in {int, float}:
return True if math.isclose(a, b, rel_tol=1e-05) else a <= b # default: rel_tol=1e-09
else: # date
return a <= bI don't recall the rationale here, but it might have been motivated by the bin validation done by _validate_bin_prediction_dict():
BIN_SUM_REL_TOL = 0.001 # hard-coded magic number for prediction probability sums
def _validate_bin_prediction_dict(is_validate_cats, prediction_dict, target):
...
prob_sum = sum(prediction_data['prob'])
if not math.isclose(1.0, prob_sum, rel_tol=BIN_SUM_REL_TOL):
raise RuntimeError(f"For one prediction element, the values within prob must sum to 1.0. "
f"prob_sum={prob_sum}, delta={abs(1 - prob_sum)}, rel_tol={BIN_SUM_REL_TOL}, "
f"prediction_dict={prediction_dict}")
nickreich
lshandross
When uploading forecasts in which quantile values are small (e.g. 1.75e-05), zoltar throws the error message "Entries in
valuemust be non-decreasing as quantiles increase" even when this is not true.For example, see the 2022-11-14 forecast for the VTSanghani-Transformer model, original forecast CSV attached. Attempting to upload a transformed JSON file of the forecasts to zoltar (see code below) results in a job output listed here. Both the original CSV file and transformed JSON file have the correct values which are non-decreasing as the quantiles increase, in spite of the error message thrown by zoltar.
Note that in spite of the error message shown above this forecast passed a sanity check validation on the original GitHub repo here, which I would think includes ensuring there are no decreasing values for increasing quantiles
2022-11-14-VTSanghani-Transformer.csv cdc.txt