densenet and shufflenetv1 model low accuracy

[aceduce]

Hi, I ran most CNN models in the model zoo for onnx and achieve similar/reasonable accuracy for the imagenet test data.

1. but when it comes to densenet and shufflenet, I saw the accuracy is very very low. For ShuffleNet, I always observe same results labels almost regardless of what input images I give. This is tested under Caffe2, (and MXNET, TVM, when feasible),. They all show similar issues. Is it because some preprocessing is missed? in the doc, it doesn't say any preprocessing is needed. Or it's due to some model issues.

2. As some previous issues reported (https://github.com/onnx/models/issues/82), models such as resnet in onnx/models is not as good as in onnx/models/models/image_classification/, what's the cause for that? in general, I don't see preprocessing specified in models under onnx/models. I also observe point 3 as follows in Pytorch, which could be a hint, I guess (?)

3. It seems in Pytorch torchvision, there are base models (such as resnet, densenet, squeezenet) and related specific models (such as resnet18, densenet121 etc), I think they are the ones exported to onnx/models and /models/models/image_classification/ respectively. Does anyone know the background why there are separate models in torchvision? Are the 'base' model ready for use also, or just the specific models? Through the test I did, I find out that most models under https://github.com/onnx/models/tree/master/models/image_classification can reach the good level of accuracy.

Just to recap, the core questions are how to deploy the densenet and shufflenet under onnx/models/ to reasonable accuracy. Please help!

Thank you!

[ahmedhosny]

I hope we can revisit this one. I can confirm getting similar results for ShuffleNet with mxnet backend: it seems stuck on the same results regardless of input. I tried normalizing, centering, and standardizing the input with no avail. The same is true for all 4 versions of shufflenet.onnx. It outputs the following 2 labels for almost all images I tested with:

"401": "accordion, piano accordion, squeeze box"
"904": "window screen"

Here is a minimal code snippet using mxnet==1.3.1 --pre -U :

from PIL import Image
import numpy as np
import mxnet as mx
from collections import namedtuple
from mxnet.contrib.onnx import import_model
import mxnet.contrib.onnx as onnx_mxnet

image_file = "###"
model_file = "###"

image = Image.open(image_file)
image = image.resize((224,224), resample = Image.LANCZOS)
arr = np.array(image).reshape(1, 3,224,224)
ctx = mx.cpu()
sym, arg, aux = onnx_mxnet.import_model(model_file)
data_names = [graph_input for graph_input in sym.list_inputs()
                      if graph_input not in arg and graph_input not in aux]

mod = mx.mod.Module(symbol=sym, data_names=data_names, context=ctx, label_names=None)
mod.bind(for_training=False, data_shapes=[(data_names[0], arr.shape)], label_shapes=None)
mod.set_params(arg_params=arg, aux_params=aux, allow_missing=True, allow_extra=True)

batch = namedtuple('Batch', ['data'])
mod.forward(batch([mx.nd.array(arr)]), is_train=False)
prob = mod.get_outputs()[0][0].asnumpy()
print (np.argmax(prob), np.max(prob))

Has #77 been addressed?

[ankkhedia]

@bddppq @houseroad Could you please take a look into the same ?