search

mit-han-lab / mcunet

[NeurIPS 2020] MCUNet: Tiny Deep Learning on IoT Devices; [NeurIPS 2021] MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
https://mcunet.mit.edu
MIT License
426 stars 79 forks source link

Clarification of ImageNet dataset used #12

Open MiguelCosta94 opened 1 year ago

MiguelCosta94 commented 1 year ago

Hi,

I am trying to evaluate the mcunet-320kB implemented in TFLite using your eval_tflite.py script. I followed the steps provided in this Git issue ( https://github.com/mit-han-lab/mcunet/issues/11 ), but I am still getting a very low accuracy of 0.08%. I downloaded the 2012 version of the ImageNet dataset from this link https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar and have prepared the repo using this script https://github.com/pytorch/examples/blob/main/imagenet/extract_ILSVRC.sh . I am doing something wrong?

tonylins commented 1 year ago

Hi, could you please check if you can reproduce the ImageNet accuracy number of pre-trained torchvision models with the official PyTorch evaluation script? Thanks!

tonylins commented 1 year ago

We followed the procedures here to process the dataset: https://github.com/pytorch/examples/tree/main/imagenet

MiguelCosta94 commented 1 year ago

I was already able to reproduce the results from the paper. They are now very close to the reported accuracy +-2%. Thank you so much.