Downgrade of ~10% on multiple weights of YoloX

[eladmeir]

Issue Type

Others

OS

Linux

onnx2tf version number

1.17.5

onnx version number

1.14.1

onnxruntime version number

1.16.0

onnxsim (onnx_simplifier) version number

0.4.33

tensorflow version number

2.13.1

Download URL for ONNX

https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_s.onnx

Parameter Replacement JSON

No need

Description

1. Research
2. I have tested 4 different models (same architecture, different data X 4), and all of them are leading to same downgrade of accuracy when comparing the onnx and .pb files - the onnx give me ~10% better accuracy than the .pb one
3. I have looked at the logging and everything seems fine - all of the single layer's comparisons gave exact results, and there is no difference with any layer
4. Currently I cannot use my .pb model on my TF1 server, because it is not good enough (the onnx was indeed good enough)
5. https://github.com/Megvii-BaseDetection/YOLOX - the source for the onnx model

[PINTO0309]

pip show onnx2tf
Name: onnx2tf
Version: 1.18.14

onnx2tf -i yolox_s.onnx -cotof

With the understanding that you are looking at the same conversion log as I am, the output is an exact match. It must be a problem with your logic or TensorFlow v1.

dup: https://github.com/PINTO0309/onnx2tf/issues/507

Frankly, I don't recommend the Protocol Buffer, and I don't think Google is nearly willing to do it either.

[eladmeir]

Thanks for your quick reply

I wan to point out that I am using TF 2.13, so maybe the "TF<2.10.0" tag is irrelevant Also - I do not think that this is a duplicate of #507 due to the specific code that both of us (that was me on #507 :) ) added and is not available anywhere on this repository. It is a very complex set of lines that without them - one cannot make a good usage of the .pb model.

As for your tip - Protocol Buffer is being used in TF by default, so basically you are suggesting to not use .pb models at all? I was not aware that PBs were not a good practice, could you elaborate and maybe point out to a better solution for TF?

[PINTO0309]

I wan to point out that I am using TF 2.13, so maybe the "TF<2.10.0" tag is irrelevant

The .pb of TF v1.x and the saved_model.pb of TF v2.x have distinctly different specifications. For example, in v1.x .pb, differences in the parameters available for Resize and bugs in TensorFlow's internal implementation are still present. As a splice, Keras is forcibly merged and forcibly redirected to V1 logic via compat.v1.

In other words, we are aware that no maintenance (including bugs in the specification) has been done on the parts of the system where the V1-based internal logic is called.

I'm not going to go back and read TensorFlow code that is two to three years old logic and try to solve the problem.

https://github.com/tensorflow/tensorflow/tree/v1.15.5

[eladmeir]

Oh, I get what you are saying... Thanks for the explanation.

Maybe I will try and see if I could replace my TF V1.x server to something more stable

And once again - thanks for the wonderful work that you are doing here.

[github-actions[bot]]

If there is no activity within the next two days, this issue will be closed automatically.

[eladmeir]

I HAVE FOUND A BUG IN MY CODE, IT WAS A FALSE ALARM

Just a quick update, for the sake of future usage - I have found a tiny (but major, obviously) bug on my python NMS, that was replacing the original torch implementation of NMS from the original YoloX repository

After fixing the bug - I was able to assert that the overall YoloX .pth -> .onnx -> .pb (TF1.X format) is almost perfect, meaning identical up to a tiny fraction of an error between the 3 models (the error is negligible for most scenarios, and is almost zero when testing on multiple scenarios of model weights and benchmarks)

Thanks for the support

[PINTO0309]

Excellent. Thank you for sharing.