search

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks
https://mlcommons.org/en/groups/inference
Apache License 2.0
1.15k stars 508 forks source link

WG decision on Edge scenario Object detector #1110

Open rnaidu02 opened 2 years ago

rnaidu02 commented 2 years ago

Retinanet-ResNeXt50 with 800x800 will be the candidate for Object Detector in v2.1 for Data Center. What would be the equivalent for Edge? (up to v2.0, SSD-Mobilenet 300x300 for Edge and SSD-ResNet34 1200x1200 for Data Center)

Decide on next steps:

  1. Poll with option for Edge?
YungbumJung commented 2 years ago

FuriosaAI suggests model EfficientDet for Object Detector for Edge.

Many models are optimized for NVIDIA GPUs because the current training market is dominated by NVIDIA. This is no exception for the models used in MLPerf. MLPerf therefore still struggles to attract wider participation from accelerator suppliers. A founder of a major AI research firm felt "most companies don't see enough ROI from the massive effort required to run these benchmarks". This is definitely not ideal, in view of the spirit of MLPerf, which should encourage a diverse spectrum of suppliers to participate.

We do agree that SSD-small is an old model, with little practical application value - so we do not object to removing this model from the benchmark. Instead, in stimulating attempts towards new architectures that are different from existing NVIDIA optimized models, we suggest adding one of EfficientDet-D0/D1/D2 as an Object Detection model. Compared to RetinaNet, EfficientDet requires less computations but shows comparable accuracy levels. This should be appropriate for AI accelerators to work on.