search

nod-ai / SHARK-Turbine

Unified compiler/runtime for interfacing with PyTorch Dynamo.
Apache License 2.0
82 stars 41 forks source link

[Tracker] Onnx FE Support #564

Open vivekkhandelwal1 opened 3 months ago

vivekkhandelwal1 commented 3 months ago

This issue is for the purpose of tracking all the ONNX Frontend Requirements, categorized into following parts (listed in the highest to lowest priority order):

Models: 1.Vision Models : 76 int8 quantized CNN Models used for vision 2.P0/P1 CNN Models : 487 int8 IPU CNN Models 3.Protected Models : 23+ int8 Models in the protected area (ISV or benchmark protected models) 4.Hugging Face CNN Models : 900 fp32 hugging face CNN models 5.Hugging Face non-CNN Models : Around 110 hugging face non-CNN models

IREE EP Support:

  1. Run a sampled set of tests (20 tests) from above by loading in IREE Execution Provider, IREE EP on Windows 11 and Ubuntu 22.04.

OP Support and Tests:

  1. OP Suport ONNX OP support to lower to linalg from ONNX
  2. Lit Tests of torch-mlir (ONNX to linalg lowering)
  3. ONNX Op Shark_TestSuite/iree_tests
  4. Torch Op E2E Tests of torch-mlir

Latest Status (Passing/Total)

Item Status Owner July 1 Prev (July 1) Current (July 8) Target (Sept 30 )
Vision Models @AmosLewis 45/63 (72%) 45/63 (72%) TBD 76/76 (100%)
P0/P1 CNN Models @AmosLewis ?/487 ?/487 TBD 487/487 (100%)
Protected Models @zjgarvey 2/23 (9%) 2/23 (9%) TBD 23/23 (100%)
Hugging Face CNN Models @jinchen62 ? /900 ? /900 TBD 900/900 (100%)
Hugging face non-CNN models @renxida ?/110 ?/110 TBD N/N (100%)
IREE EP Models test @Shukla-Gaurav ?/20 ?/20 TBD K/K (100%)
OP Support @vivekkhandelwal1 177/193 (92%) 177/193 (92%) TBD 100%
Lit Tests @renxida 332/433(77%) 332/433(77%) TBD 100%
Onnx Op @jinchen62 640/1234 (52%) 640/1234 (52%) TBD 65%
Torch OP @zjgarvey 838/1363 61% 838/1363 61% TBD 65%

Note: Just a gentle request that if you're working on fixing any of these failures then please go and check the other kind of failures related to the issue you're working on. Your fix for one kind of failure might fix the other kind of failure too. For Models, if iree-compile passes, count as pass. We will require inference to pass once @saienduri has done his work.

Lit Tests of torch-mlir (ONNX to linalg lowering)

Owner: @renxida

The Onnx lowering Lit Tests are tracked through https://github.com/nod-ai/SHARK-Turbine/issues/450.

Steps to reproduce:

View the op name from the tracker and then take out the lit test corresponding to that op in a seperate file, and run:

torch-mlir-opt --convert-torch-onnx-to-torch --torch-decompose-complex-ops --canonicalize --torch-backend-to-linalg-on-tensors-backend-pipeline test.mlir

Torch Op E2E Tests of torch-mlir

Owner: @vivekkhandelwal1 @zjgarvey

The Torch-MLIR Op E2E Tests are tracked through https://github.com/nod-ai/SHARK-Turbine/issues/549

Steps to reproduce:

Take out the E2E test from the tracker and run:

python -m projects.pt1.e2e_testing.main -f <test_name> -v --config=onnx

ONNX Op Shark_TestSuite/iree_tests

Owner: @jinchen62 @Shukla-Gaurav

The Shark_TestSuite IREE Tests - ONNX Op tests are tracked by Compile time Tests - https://github.com/nod-ai/SHARK-Turbine/issues/563 Runtime Tests - https://github.com/nod-ai/SHARK-Turbine/issues/583

Steps to reproduce:

To run the test, please follow: build venv following here and run 

iree-compile` iree_tests/onnx/node/generated/TEST_NAME/model.mlir -o test.vmfb --iree-hal-target-backends=llvm-cpu --mlir-print-ir-after-all

Models Shark_TestSuite/e2eshark

Owner: @PhaneeshB @saienduri @AmosLewis

The E2EShark Model Tests are tracked through https://github.com/nod-ai/SHARK-Turbine/issues/566

Steps to reproduce:

First, follow setup instructions at https://github.com/nod-ai/SHARK-TestSuite/tree/main/e2eshark. No need to do the Turbine setup part as we are looking at onnx mode. Then, run this command (HF_TOKEN needed for llama, gemma model):

HF_TOKEN=your_token python run.py --torchmlirbuild /path/to/torch-mlir/build --ireebuild /path/to/iree-build --cachedir ~/.cache/huggingface --tests pytorch/models/<model_name> -r test-onnx --tolerance .001 .001 --mode onnx --report
kumardeepakamd commented 3 months ago

Owners, kindly add clear steps to reproduce failures and allow ability for contributors to take up a unique issue and work on fix to have more folks join in for this quality push. Great start! Let's do it.