Open saienduri opened 6 months ago
Please add Path to the logs directory to make it clear where to look for them. For model
$ SHARK-TestSuite/e2eshark/test-onnx/pytorch/models/<model_name>/model-run.log
$ SHARK-TestSuite/e2eshark/test-onnx/pytorch/models/<model_name>/iree-compile.log
Working on the "Add" Issue. Please assign: https://github.com/nod-ai/SHARK-Turbine/issues/586
A regression on 2024-04-30 https://github.com/nod-ai/e2eshark-reports/blob/main/2024-04-30/onnx_reports/statusreport.md 2024-04-18 | pytorch/models/gpt2-xl | passed | passed | passed | passed | passed | | pytorch/models/resnet50 | passed | passed | passed | passed | passed | 2024-04-30 | pytorch/models/gpt2-xl | passed | passed | notrun | failed | notrun | | pytorch/models/resnet50 | failed | notrun | notrun | notrun | notrun |
I'm not sure what is the cause of the discrepancy with the current list of issues, but with an up-to-date torch-mlir with a few minor edits to the recent work in fuse quantized ops, here's a triage list for torch-mlir failures when running:
python run.py --cachedir="/home/zjgar/.cache/" --torchtolinalg -c "/home/zjgar/code/torch-mlir/build/" --mode=onnx --groups=models --framework=onnx
list of failures (and a brief triage):
Test onnx/models/VideoResNet_vaiq_int8 failed [torch-mlir]
onnx.constant??
Test onnx/models/MobileNetV3_small_vaiq_int8 failed [torch-mlir]
grouped q convolution
Test onnx/models/RegNet_y_8gf_vaiq_int8 failed [torch-mlir]
grouped q convolution
Test onnx/models/Inception_v4_vaiq_int8 failed [torch-mlir]
average Pool
Test onnx/models/pytorch-3dunet_vaiq_int8 failed [torch-mlir]
resize
Test onnx/models/ShuffleNet_v2_x2_0_vaiq_int8 failed [torch-mlir]
grouped q convolution
Test onnx/models/MNASNet_1_3_vaiq_int8 failed [torch-mlir]
grouped q convolution
Test onnx/models/LRASPP_vaiq_int8 failed [torch-mlir]
grouped q convolution
Test onnx/models/RRDB_ESRGAN_vaiq_int8 failed [torch-mlir]
resize
Test onnx/models/KeypointRCNN_vaiq_int8 failed [torch-mlir]
onnx if
Test onnx/models/EfficientNet_v2_s_vaiq_int8 failed [torch-mlir]
grouped q convolution
Test onnx/models/retinanet_resnet50_fpn_vaiq_int8 failed [torch-mlir]
onnx if
Test onnx/models/ConvNeXt_vaiq_int8 failed [torch-mlir]
grouped q convolution
Ty! On it. Will dive deeper tomorrow.
Also, when posting commands, will so love it if you could do
python run.py --cachedir="~/.cache/" --torchtolinalg -c "~/torch-mlir/build/" --mode=onnx --groups=models --framework=onnx
I think a sizeable chunk have each repo directly cloned to our home dir so something like this would be directly runnable.
Test onnx/models/pytorch-3dunet_vaiq_int8 failed [torch-mlir] resize @aldesilv
Test onnx/models/RRDB_ESRGAN_vaiq_int8 failed [torch-mlir] resize @zjgarvey
Test onnx/models/pytorch-3dunet_vaiq_int8 failed [torch-mlir] resize @aldesilv
the immediate issue is the dynamic dims in the input torch.vtensor<[?,256,?,?,?],f32> checked causing the compile error. Next would be the 3-d input
3 pytorch model failed again from 2024-05-29 to 2024-05-30 e2eshark-reports.
| pytorch/models/bert-large-uncased | passed | passed | notrun | failed | notrun |
| pytorch/models/bge-base-en-v1.5 | passed | passed | notrun | failed | notrun |
| pytorch/models/miniLM-L12-H384-uncased | passed | passed | notrun | failed | notrun |
3 pytorch model failed again from 2024-05-29 to 2024-05-30 e2eshark-reports.
| pytorch/models/bert-large-uncased | passed | passed | notrun | failed | notrun | | pytorch/models/bge-base-en-v1.5 | passed | passed | notrun | failed | notrun | | pytorch/models/miniLM-L12-H384-uncased | passed | passed | notrun | failed | notrun |
Any idea what they are failing on?
Any idea what they are failing on?
Not sure, working with @saienduri to figure it out. I just tested with 0530 torch-mlir d7b8f00 and iree candidate-20240530.909 locally, they passed. It's kind of weird. Sai think it might pass with latest iree, let's see what's going on with 0531 report.
Any idea what they are failing on?
Not sure, working with @saienduri to figure it out.
We have rooted the 3 model regression to 40 passes in https://github.com/nod-ai/e2eshark-reports/tree/main/2024-05-31 to the convert-torch-onnx-to-torch pass being outdated in iree (generating different mlirs compared to torch-mlir TOM). So, once torch-mlir gets bumped in iree, they should pass again :)
Below is the list of issues we are hitting when running vision int8 models end to end using onnx mode (onnx export/import -> torch-mlir -> iree-compile -> iree-runtime). You can find the models that lead to each issue in the issue description.
To reproduce the error, please setup SHARK-TestSuite and then run the run.py file with the respective command line flags (More Instructions can be found here.)
To fix the issue, you need to either modify the OnnxToTorch lowering of the corresponding op or add the missing support in the TorchToLinalg lowering. You can find more information in either the model-run.log or iree-compile.log after running the test. This can help you create a smaller repro and then try to fix that, then check if it fixes the model.
You can find the specific logs on what is failing in these locations for where SHARK-TestSuite/e2eshark/test-onnx is test run directory:
Issues:
torch-to-linalg
iree:
Onnx VAIQ Models
To run all tests :
python run.py --torchmlirbuild /path_to/torch-mlir/build --ireebuild /path-to/iree-build --cachedir /path-to/model-cache-dir -r test-onnx --tolerance .001 .001 --mode onnx --report -f onnx -g models
To run specific test (ex: onnx/models/AlexNet_vaiq_int8)
python run.py --torchmlirbuild /path_to/torch-mlir/build --ireebuild /path-to/iree-build --cachedir /path-to/model-cache-dir -r test-onnx --tolerance .001 .001 --mode onnx --report --tests onnx/models/AlexNet_vaiq_int8
Versions:
torch-mlir
- main -a7302a68
iree
- main -40f25334d2
Status: Check the latest run report in e2eshark-reports :
e2eshark-reports/<DATE>/onnx_reports/statusreport.md
onnx model Pass(with --torchtolinalg):28/34, Day:08/08
pytorch model , Pass(with --torchtolinalg): 4/17/28, Day: 08/08