[Web] Inconsistent results between running onnx model through python and with onnxruntime-web

[mr-sarthakgupta]

Describe the issue

I'm trying to run this model in the browser using onnxruntime-web but the results aren't consistent when compared to running the model using python.

I'm observing the following results:

torch vs onnx cosine similarity: 0.9999999999743225
torch vs transformersjs local cosine similarity: 0.7655750265496857
torch vs ortw firefox similarity: 0.7699873570121704
torch vs ortw chrome similarity: 0.7984060531122313
ortw firefox vs ortw chrome similarity: 0.8741733200597323
torch vs transformersjs browser cosine similarity: 0.7689434867346924

Please note that transformersjs also uses an onnx runtime to do the model passes. This is severely affecting model performance and delaying a project.

To reproduce

I'm using the latest versions of onnxruntime-web and onnx. I converted the model from pytorch to onnx using huggingface optimum library. Since the onnx and pytorch models agree anyways so the computational graph of the model is exported correctly. I am manually verifying that the inputs in all cases are practically the same, using the same preprocessing steps everywhere.

Urgency

The issue is urgent as there's a project deadline quick approaching which requires this inconsistency to be resolved. I've already spent time trying to dig deeper into this issue wrt the transformersjs library but it seems to be an issue with the onnxruntime(transformersjs also uses an onnx runtime).

Please help me getting this resolved as quickly as possible, I'm really hopeful about this.

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.18.0

Execution Provider

'wasm'/'cpu' (WebAssembly CPU)

[mr-sarthakgupta]

@snnn please help

[Sedherthe]

Hey @mr-sarthakgupta, is your issue resolved?

[mr-sarthakgupta]

Unfortunately no @Sedherthe

[github-actions[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[gyagp]

I compared the data between onnxruntime wasm ep and webgpu ep, and their results are very similar. Below are the results from 3 executions (random input with shape [1, 3, 224, 224]). Can you share your python code and inputs?