Missing results in metrics

[geoalgo]

We talk offline about this but I wanted to create an issue so that we dont forget this one.

Currently, for rare cases we have tabular predictions available but not metrics (such as runtime).

from autogluon_zeroshot.repository.evaluation_repository import load

repo = load(version="BAG_D244_F10_C608_FULL")
config = "RandomForest_r6_BAG_L1"
tid = 3483
fold = 0

# prediction available in tabular predictions...
print(repo._tabular_predictions.predict(tid, 0, models=[config]))

# ... but not in results
dd = repo._zeroshot_context.df_results_by_dataset_vs_automl
print(len(dd[(dd.framework == config) & (dd.dataset == f"{tid}_{fold}")]))

# print missing tasks
xx = dd.pivot_table(index="framework", columns="dataset", values="score_val").loc["RandomForest_r6_BAG_L1"]
print(xx[xx.isna()])

We could backfill metrics (ideal), in the meantime we could also impute metrics or remove the partial tasks, currently the following tasks are not complete for "RandomForest_r6_BAG_L1:

3483_0     NaN
3583_0     NaN
359932_0   NaN
359932_8   NaN
359933_8   NaN
359944_1   NaN
359944_8   NaN
359944_9   NaN
359946_0   NaN
359946_2   NaN
359946_5   NaN
58_9       NaN

[geoalgo]

Resolving as we are now imputing this data and regenerating a dataset.