search

facebookresearch / dynalab

The Python library with command line tools to interact with Dynabench(https://dynabench.org/), such as uploading models.
MIT License
54 stars 10 forks source link

Improved Reporting of Model Upload Status #85

Open maxbartolo opened 3 years ago

maxbartolo commented 3 years ago

The model upload process is pretty straightforward and user-friendly. That said, some feedback in addition to the status would be helpful. For example:

  1. A "publishable" status indicating the status of all the steps remaining for the model to appear on the leaderboard e.g. fairness testing, robustness testing, etc
  2. A per-dataset status (especially as the number of datasets scale, for example, model https://dynabench.org/models/109 took approx. 16hrs for all the non-leaderboard dataset evaluations to come in). This could include statuses such as "Queued", "Running", "Complete", "Failed", etc
mazhiyi commented 3 years ago

Noted. Thanks Max! [internal] This issue should be moved to dynabench