Open rstojnic opened 4 years ago
In some cases the first batch might be too small, or simply contain no interesting predictions (eg empty or all zeros).
Run hash calculation should catch this, produce a warning and not try to retrieve the cached results.
This might be an issue for the frontends (sotabencheval, torchbench) and is task-specific.
I've solved for COCO in sotabencheval by doing the following:
In some cases the first batch might be too small, or simply contain no interesting predictions (eg empty or all zeros).
Run hash calculation should catch this, produce a warning and not try to retrieve the cached results.