Evaluated classifications onnx models almost with zero accuracy

onnx / models

A collection of pre-trained, state-of-the-art models in the ONNX format

http://onnx.ai/models/

Apache License 2.0

7.46k stars 1.36k forks source link

Evaluated classifications onnx models almost with zero accuracy #622

Open Apisteftos opened 11 months ago

Apisteftos commented 11 months ago

I evaluated all of the classifications models according to their preprocessing description with imagenet:

Models:

squeezenet1.0-12.onnx
bvlcalexnet-12.onnx
caffenet-12.onnx
rcnn-ilsvrc13-9.onnx
inception-v1-12.onnx
inception-v2-9.onnx
zfnet512-12.onnx

Why should this models being in the model zoo if their behavior and accuracy is terrible?

gcunhase commented 5 months ago

Any updates?

gcunhase commented 5 months ago

@mengniwang95 @bddppq @houseroad

mengniwang95 commented 5 months ago

Hi, there are many different versions for the same model and they are from more than one contributors. We submitted part of them previously and verified the accuracy with our tools. Since there are others contributions, we only add our instruction and keep others' instructions. You can find the details of how to reproduce the result step by step in https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/image_recognition/onnx_model_zoo, https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/object_detection/onnx_model_zoo, https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/body_analysis/onnx_model_zoo and https://github.com/intel/neural-compressor/tree/master/examples/onnxrt/nlp/onnx_model_zoo.

gcunhase commented 5 months ago

Hi @mengniwang95 thank you for your feedback. I tried following the pre-processing in the links you sent but that was either non-existent or didn't work. Could you please verify a model with issue such as inception-v1-12? According to the table here, the accuracy for that model should be 67.23% but I'm getting close to 0%. Thank you.