[Announce] Next generation of ONNX Model Zoo

[jcwchen]

Feature Request

What is the problem that this feature solves?

• All of existing ONNX Model Zoo models use opset_version<13, which is a quite old version (opset 12 released on May 8, 2020 in ONNX 1.7.0).
• Existing models are hard to reproduce with outdated script ​
• Same model usually has few versions​
• No sufficient state-of-art models in the past 2 years

Describe the feature

• Introduce new upload way with required reproduction files: https://github.com/onnx/models/pull/613
• Move existing models (< opset_version 13) to models/legacy: https://github.com/onnx/models/issues/616
• Add a few existing models from torch_hub or torchvision with opset_version ~18~ 16: https://github.com/onnx/models/pull/614
• Add more state-of-the-art models with opset_version ~18~ 16: https://github.com/onnx/models/pull/615
• Utilize benchmark tool (MLAgility) to verify uploaded models
• Keep single version of model and going forward ONNX Model Zoo will regularly bump opset_version
• New models directory hierarchy:
  • models/python: user-contribution-based models. There will be models along with reproduction script to ensure the models are reproducible.
  • models/mlagility: only contains models and test_data_set; reproduction script can be found at https://github.com/groq/mlagility/tree/main/models.
  • models/legacy: old models with opset_version < 13
• ONNX Hub will support download models from the three source above
• Web interface to view ONNX Model Zoo models
• Eventually we will remove models under legacy

Relevant Model Zoo feature areas

Which area in the Model Zoo infrastructure does this impact?
Feature Area (e.g. CI, file organization, model storage, other): Model storage/CI

Notes

Currently this feature is working in progress and we are developing in the "new-models" branch first. Ultimately we will move this branch into the main branch.

If anyone has any question/concern, feel free to raise in this thread. Thank you for the contribution.

cc @abock @ramkrishna2910

[chenqiny]

May I know where can I find the test datasets?

In old version, there was datasets for each model.

[jcwchen]

May I know where can I find the test datasets?

@ramkrishna2910 do we have a future plan to add test datasets for recently brought models? If no, is there any way to let users create test data from their own?