search

mlcommons / tiny

MLPerf™ Tiny is an ML benchmark suite for extremely low-power systems such as microcontrollers
https://mlcommons.org/en/groups/inference-tiny/
Apache License 2.0
331 stars 81 forks source link

Inconsistency with bias enabled in the pytorch model (Conv2d) for image classification #147

Open embrya-technologies opened 5 months ago

embrya-technologies commented 5 months ago

Hello,

I am interested in running the image classification model to benchmark our accelerator and, currently, my environment is in pytorch therefore, I had a look at your experimental model under: /tiny/benchmark/experimental/training_torch/image_classification/utils/model.py

The model contains ResNetBlock with two Conv2d convolutions followed per Batch Normalization. However, each Conv2d layer is configured with the bias enabled (bias=True) which is inconsistent with the Con2d layers in the Keras model that don't have the use_bias flag enabled (also it is not coherent with the purpose of batch normalization layer that follows).

Thank you, Best regards, Jean-Baptiste

cskiraly commented 5 months ago

Thanks for opening the issue @embrya-technologies !

Just to be clear for someone stumbling on this issue, as stated in the README, the PyTorch training code is not official and cannot be used for closed submissions. Even the original training code is only for documentation purposes. The only official source for closed submissions is the trained model itself.

Having said that, it is important to make this code map as closely as possible. Thanks again for reporting it! It would be great if you could make a PR to fix the issue.