paperswithcode / sotabench-api

Easily benchmark Machine Learning models on selected tasks and datasets
Apache License 2.0
15 stars 5 forks source link

Run hash should give a warning for empty/all-zero inputs #18

Open rstojnic opened 4 years ago

rstojnic commented 4 years ago

In some cases the first batch might be too small, or simply contain no interesting predictions (eg empty or all zeros).

Run hash calculation should catch this, produce a warning and not try to retrieve the cached results.

RJT1990 commented 4 years ago

This might be an issue for the frontends (sotabencheval, torchbench) and is task-specific.

I've solved for COCO in sotabencheval by doing the following: