Converted model outputs are not as expected

Roberto7766 commented 3 years ago

1. OS you are using e.g. Ubuntu 20.04, WIndows10, etc

Colab

2. OS Architecture e.g. x86_64, armv7l, aarch64, etc

Colab

3. Version of OpenVINO e.g. 2021.2.185, etc

openvino_2021.1.110

4. Version of TensorFlow e.g. v2.4.1, tf-nightly==2.5.0.dev20210128, etc

tensorflow-2.4.1

5. Version of TensorRT e.g. TensorRT6.0 GA, etc

6. Version of TFJS e.g. 1.5.0, etc

7. Version of coremltools e.g. 4.0, etc

8. Version of ONNX e.g. v1.8.0, etc

onnx-1.8.1

9. Download URL for ONNX model

!pip install onnx

13. Issue Details

Hi, thanks for the really great work! I'm sharing my Colab notebook with all the steps to replicate the issue. Basically what I obtain is a converted model that is quite similar to the original ONNX model but not really similar, in the sense that using the same inputs, the outputs are slightly different. Using different input pattern, the difference can be even higher. Am I doing something wrong? Thanks.

parthjdoshi commented 3 years ago

Hey @PINTO0309 Any updates on this?

PINTO0309 commented 3 years ago

osnet.tar.gz

PINTO0309 commented 3 years ago

Or, If you suspect a bug in my tool or in the OpenVINO optimizer, you can use onnx-tensorflow.

onnx-tf convert -i osnet_x0_25_msmt17_1.onnx -o saved_model

PINTO0309 commented 3 years ago

osnet_x0_25_msmt17_1.onnx.tar.gz

[[6.6613150e-01 0.0000000e+00 7.0128053e-01 1.4839412e+00 0.0000000e+00
  1.4773365e+00 0.0000000e+00 0.0000000e+00 9.7492540e-01 0.0000000e+00
  5.6594663e+00 1.7946564e+00 0.0000000e+00 0.0000000e+00 5.4976773e-01
  3.9135149e+00 0.0000000e+00 0.0000000e+00 8.5506839e-01 0.0000000e+00
  7.3580778e-01 0.0000000e+00 0.0000000e+00 5.1075389e-05 1.3429124e-06
  0.0000000e+00 0.0000000e+00 4.7461498e-01 0.0000000e+00 0.0000000e+00
  4.2130755e-06 1.1735106e+00 0.0000000e+00 0.0000000e+00 1.3936181e+00
  0.0000000e+00 0.0000000e+00 4.1948799e-05 0.0000000e+00 0.0000000e+00
  2.1539381e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 3.5236311e+00
  4.4628927e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.4151138e+00
  0.0000000e+00 0.0000000e+00 7.8680450e-01 0.0000000e+00 0.0000000e+00
  2.7677208e-02 4.3038261e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 4.3686442e+00 4.2978936e-01 3.1850669e-07
  3.2601297e+00 1.4938724e-01 0.0000000e+00 0.0000000e+00 7.2955054e-01
  3.8973013e-01 0.0000000e+00 0.0000000e+00 4.6151054e-01 2.0457351e+00
  7.3345882e-01 4.0035212e-01 7.6619452e-01 0.0000000e+00 0.0000000e+00
  9.1156326e-06 0.0000000e+00 5.8929741e-01 4.6722851e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 6.3894027e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 4.7993593e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.0735338e+00
  6.8698235e-08 0.0000000e+00 0.0000000e+00 6.2981873e+00 0.0000000e+00
  9.7945607e-01 1.4582504e-01 0.0000000e+00 1.4289470e+00 1.0721380e-02
  0.0000000e+00 1.6833425e-07 3.9504466e+00 1.3682890e-01 0.0000000e+00
  3.3840866e+00 0.0000000e+00 2.8281345e+00 5.0573339e+00 1.4943421e+00
  5.4386970e-07 3.3179679e+00 2.5023360e+00 1.9066582e+00 8.1628857e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 2.7667612e-01 1.3142862e+00
  7.8361374e-08 2.4787221e+00 8.8061333e-01 0.0000000e+00 0.0000000e+00
  5.6407458e-01 0.0000000e+00 5.5099225e-01 0.0000000e+00 3.6925416e+00
  2.4703169e-01 0.0000000e+00 6.5545839e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 7.4483621e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 1.5538825e-01 5.6379110e-02 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 3.4847796e-01 7.7743864e-01
  0.0000000e+00 3.7505490e-01 1.4341450e+00 2.7678270e+00 1.3692247e-03
  0.0000000e+00 9.4777024e-01 0.0000000e+00 3.3196920e-01 0.0000000e+00
  0.0000000e+00 3.3061335e-06 0.0000000e+00 0.0000000e+00 0.0000000e+00
  1.8760824e-01 0.0000000e+00 0.0000000e+00 2.4425244e+00 1.8193296e+01
  0.0000000e+00 0.0000000e+00 1.6534125e+00 0.0000000e+00 3.2500029e-01
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 5.1153898e+00
  2.3521953e+00 0.0000000e+00 4.8439634e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 1.3338089e-06 5.3591192e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 1.6920059e+00 4.8520866e-01
  2.1923633e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 3.8131430e+00
  1.2381130e-04 1.4448266e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 3.0955064e-01 0.0000000e+00 6.3019337e-07 1.1905499e+00
  1.7286396e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 7.4661241e+00
  9.5084459e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.7788360e+00
  0.0000000e+00 2.4178939e+00 1.1731247e+00 4.0737948e+00 1.5062264e-01
  0.0000000e+00 0.0000000e+00 0.0000000e+00 8.9462483e-06 0.0000000e+00
  0.0000000e+00 2.6616979e+00 3.8967144e-01 7.3488851e-09 0.0000000e+00
  0.0000000e+00 3.4147186e+00 3.1662836e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 6.6708684e-02 1.5529888e+00 0.0000000e+00
  1.9119940e+00 1.0959823e+00 1.1149637e+00 0.0000000e+00 4.3283463e-01
  0.0000000e+00 1.4094681e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  1.9896673e+00 2.2269351e+00 0.0000000e+00 0.0000000e+00 1.4135696e+00
  1.1216174e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 5.1104136e+00
  4.8358088e+00 0.0000000e+00 7.5161052e-01 1.2795061e-01 8.5599506e-01
  0.0000000e+00 0.0000000e+00 9.1060936e-02 0.0000000e+00 0.0000000e+00
  0.0000000e+00 5.2937400e-01 0.0000000e+00 0.0000000e+00 1.2012825e+00
  4.2523545e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 6.3579774e-01
  2.4141474e+00 7.7673817e+00 2.7721303e-06 0.0000000e+00 0.0000000e+00
  7.3188925e-01 2.2676814e-01 8.9766514e-01 0.0000000e+00 1.5246723e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 2.3099890e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.4014381e-01
  0.0000000e+00 4.9017191e-02 2.2278931e+00 2.6072692e-06 0.0000000e+00
  1.1969380e+00 0.0000000e+00 0.0000000e+00 1.4615571e-01 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 8.3996999e-01 0.0000000e+00
  1.6047140e+00 5.9460437e-01 0.0000000e+00 2.5605426e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  7.9176152e-07 7.5739181e-01 2.5349081e+00 0.0000000e+00 9.2032230e-01
  8.6256528e-01 7.4170917e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  1.0328095e+00 2.1635926e-04 1.2477663e+00 0.0000000e+00 0.0000000e+00
  2.7259855e+00 8.6530411e-01 0.0000000e+00 1.0764526e-01 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 5.8305077e-06
  2.9083679e+00 1.7006229e+00 0.0000000e+00 1.9186589e+00 0.0000000e+00
  1.5670915e+00 1.0079299e+00 0.0000000e+00 2.2184441e+00 0.0000000e+00
  1.2611208e+00 7.2006416e-01 7.5945761e-03 6.0646429e+00 0.0000000e+00
  0.0000000e+00 6.6778344e-01 3.0681629e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 3.2933834e+00 2.7211761e+00 7.9811363e+00
  3.8762206e-01 3.9034705e+00 4.0014816e-04 0.0000000e+00 3.4476461e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 1.1982036e+00 1.7981191e+00
  2.1476147e+00 0.0000000e+00 4.7293901e-03 0.0000000e+00 8.9574176e-01
  0.0000000e+00 1.8613017e-01 3.5604843e-01 0.0000000e+00 2.0577619e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 9.7725271e-05 0.0000000e+00 0.0000000e+00
  1.9383378e+00 1.2702903e+00 1.6943543e+00 2.5589776e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 2.0130162e+00 0.0000000e+00 0.0000000e+00 3.0161376e+00
  0.0000000e+00 3.4197055e-02 0.0000000e+00 0.0000000e+00 2.4700410e+00
  3.7506723e-01 1.0811476e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  5.0253950e-02 0.0000000e+00 0.0000000e+00 2.9103258e+00 0.0000000e+00
  0.0000000e+00 2.2871962e+00 3.2629061e+00 0.0000000e+00 0.0000000e+00
  4.6550180e-03 2.0010438e+00 0.0000000e+00 0.0000000e+00 1.0298252e+00
  2.4199374e+00 5.5733338e-08 1.1096496e-01 0.0000000e+00 0.0000000e+00
  1.1499217e+00 4.1019827e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 7.1491516e-01 3.8621879e-01 2.7353215e+00 3.9005709e+00
  6.5763772e-01 1.9729968e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  2.1092761e-01 0.0000000e+00 1.6921589e+00 1.1474239e+00 0.0000000e+00
  1.9543208e+00 0.0000000e+00 0.0000000e+00 1.3428208e+00 5.2838618e-01
  0.0000000e+00 0.0000000e+00 0.0000000e+00 4.1602254e-02 0.0000000e+00
  0.0000000e+00 3.8410041e-01 3.2654295e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00]]

model_float32.tflite.tar.gz

[[6.6614509e-01 0.0000000e+00 7.0128047e-01 1.4839426e+00 0.0000000e+00
  1.4773722e+00 0.0000000e+00 0.0000000e+00 9.7492993e-01 0.0000000e+00
  5.6594028e+00 1.7946588e+00 0.0000000e+00 0.0000000e+00 5.4977214e-01
  3.9134781e+00 0.0000000e+00 0.0000000e+00 8.5507846e-01 0.0000000e+00
  7.3580831e-01 0.0000000e+00 0.0000000e+00 5.1076160e-05 1.3429125e-06
  0.0000000e+00 0.0000000e+00 4.7461468e-01 0.0000000e+00 0.0000000e+00
  4.2130628e-06 1.1735182e+00 0.0000000e+00 0.0000000e+00 1.3936067e+00
  0.0000000e+00 0.0000000e+00 4.1948399e-05 0.0000000e+00 0.0000000e+00
  2.1539562e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 3.5236564e+00
  4.4629115e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.4151008e+00
  0.0000000e+00 0.0000000e+00 7.8680891e-01 0.0000000e+00 0.0000000e+00
  2.7672201e-02 4.3038738e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 4.3686285e+00 4.2979163e-01 3.1850666e-07
  3.2601385e+00 1.4936745e-01 0.0000000e+00 0.0000000e+00 7.2957504e-01
  3.8972962e-01 0.0000000e+00 0.0000000e+00 4.6151373e-01 2.0457256e+00
  7.3346031e-01 4.0035999e-01 7.6618981e-01 0.0000000e+00 0.0000000e+00
  9.1156335e-06 0.0000000e+00 5.8929127e-01 4.6723433e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 6.3893098e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 4.7993031e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.0735435e+00
  6.8698554e-08 0.0000000e+00 0.0000000e+00 6.2981834e+00 0.0000000e+00
  9.7946560e-01 1.4582404e-01 0.0000000e+00 1.4289768e+00 1.0721380e-02
  0.0000000e+00 1.6833428e-07 3.9504664e+00 1.3683343e-01 0.0000000e+00
  3.3840771e+00 0.0000000e+00 2.8281136e+00 5.0573111e+00 1.4943674e+00
  5.4387442e-07 3.3179979e+00 2.5023932e+00 1.9066503e+00 8.1629505e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 2.7665532e-01 1.3142588e+00
  7.8358553e-08 2.4786987e+00 8.8061762e-01 0.0000000e+00 0.0000000e+00
  5.6407851e-01 0.0000000e+00 5.5102396e-01 0.0000000e+00 3.6925817e+00
  2.4701166e-01 0.0000000e+00 6.5546751e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 7.4483693e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 1.5538837e-01 5.6389272e-02 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 3.4847033e-01 7.7743083e-01
  0.0000000e+00 3.7505454e-01 1.4341824e+00 2.7678037e+00 1.3692246e-03
  0.0000000e+00 9.4778478e-01 0.0000000e+00 3.3196098e-01 0.0000000e+00
  0.0000000e+00 3.3061301e-06 0.0000000e+00 0.0000000e+00 0.0000000e+00
  1.8761057e-01 0.0000000e+00 0.0000000e+00 2.4425020e+00 1.8193272e+01
  0.0000000e+00 0.0000000e+00 1.6534206e+00 0.0000000e+00 3.2500112e-01
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 5.1153593e+00
  2.3522289e+00 0.0000000e+00 4.8440602e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 1.3338077e-06 5.3589690e-01 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 1.6920002e+00 4.8517430e-01
  2.1923404e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 3.8131618e+00
  1.2381149e-04 1.4448179e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 3.0955645e-01 0.0000000e+00 6.3019149e-07 1.1905241e+00
  1.7284226e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 7.4661112e+00
  9.5083618e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.7788435e+00
  0.0000000e+00 2.4179122e+00 1.1731027e+00 4.0738039e+00 1.5062314e-01
  0.0000000e+00 0.0000000e+00 0.0000000e+00 8.9466212e-06 0.0000000e+00
  0.0000000e+00 2.6616943e+00 3.8967419e-01 7.3488859e-09 0.0000000e+00
  0.0000000e+00 3.4147444e+00 3.1662879e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 6.6722631e-02 1.5529824e+00 0.0000000e+00
  1.9120078e+00 1.0959731e+00 1.1149586e+00 0.0000000e+00 4.3286899e-01
  0.0000000e+00 1.4098901e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  1.9896731e+00 2.2269106e+00 0.0000000e+00 0.0000000e+00 1.4135350e+00
  1.1216089e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 5.1103630e+00
  4.8358212e+00 0.0000000e+00 7.5160766e-01 1.2794897e-01 8.5597932e-01
  0.0000000e+00 0.0000000e+00 9.1065764e-02 0.0000000e+00 0.0000000e+00
  0.0000000e+00 5.2937287e-01 0.0000000e+00 0.0000000e+00 1.2012749e+00
  4.2522210e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00 6.3579452e-01
  2.4141421e+00 7.7674642e+00 2.7721303e-06 0.0000000e+00 0.0000000e+00
  7.3190016e-01 2.2677147e-01 8.9767349e-01 0.0000000e+00 1.5246429e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 2.3099773e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 1.4014095e-01
  0.0000000e+00 4.9016893e-02 2.2278876e+00 2.6072732e-06 0.0000000e+00
  1.1969175e+00 0.0000000e+00 0.0000000e+00 1.4615631e-01 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 8.3997029e-01 0.0000000e+00
  1.6047052e+00 5.9460592e-01 0.0000000e+00 2.5604849e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  7.9175965e-07 7.5739157e-01 2.5348954e+00 0.0000000e+00 9.2034400e-01
  8.6251950e-01 7.4173266e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  1.0328043e+00 2.1635942e-04 1.2477653e+00 0.0000000e+00 0.0000000e+00
  2.7260234e+00 8.6530590e-01 0.0000000e+00 1.0764554e-01 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 5.8305050e-06
  2.9083900e+00 1.7006159e+00 0.0000000e+00 1.9186342e+00 0.0000000e+00
  1.5671074e+00 1.0079255e+00 0.0000000e+00 2.2184532e+00 0.0000000e+00
  1.2611215e+00 7.2007143e-01 7.5945929e-03 6.0646133e+00 0.0000000e+00
  0.0000000e+00 6.6780895e-01 3.0681801e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 3.2933881e+00 2.7211099e+00 7.9811153e+00
  3.8762546e-01 3.9034956e+00 4.0014795e-04 0.0000000e+00 3.4475982e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 1.1982331e+00 1.7981124e+00
  2.1475930e+00 0.0000000e+00 4.7565699e-03 0.0000000e+00 8.9576113e-01
  0.0000000e+00 1.8611667e-01 3.5604745e-01 0.0000000e+00 2.0577393e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 9.7725642e-05 0.0000000e+00 0.0000000e+00
  1.9383097e+00 1.2703155e+00 1.6943400e+00 2.5589771e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 2.0130079e+00 0.0000000e+00 0.0000000e+00 3.0161397e+00
  0.0000000e+00 3.4197308e-02 0.0000000e+00 0.0000000e+00 2.4700351e+00
  3.7506360e-01 1.0811495e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  5.0250553e-02 0.0000000e+00 0.0000000e+00 2.9102907e+00 0.0000000e+00
  0.0000000e+00 2.2872052e+00 3.2629209e+00 0.0000000e+00 0.0000000e+00
  4.6550366e-03 2.0011172e+00 0.0000000e+00 0.0000000e+00 1.0297673e+00
  2.4199765e+00 5.5733551e-08 1.1096455e-01 0.0000000e+00 0.0000000e+00
  1.1499289e+00 4.1015324e-01 0.0000000e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 7.1490288e-01 3.8622642e-01 2.7353182e+00 3.9005680e+00
  6.5765154e-01 1.9730308e+00 0.0000000e+00 0.0000000e+00 0.0000000e+00
  2.1095824e-01 0.0000000e+00 1.6922119e+00 1.1474481e+00 0.0000000e+00
  1.9543558e+00 0.0000000e+00 0.0000000e+00 1.3428166e+00 5.2839828e-01
  0.0000000e+00 0.0000000e+00 0.0000000e+00 4.1584194e-02 0.0000000e+00
  0.0000000e+00 3.8410729e-01 3.2654076e+00 0.0000000e+00 0.0000000e+00
  0.0000000e+00 0.0000000e+00]]

PINTO0309 commented 3 years ago

model_float32 tflite (2)

parthjdoshi commented 3 years ago

Hey @PINTO0309 Thanks for the help. I realized that my drop in accuracy was due to the full integer quantization.

PINTO0309 / openvino2tensorflow