search

nod-ai / SHARK-TestSuite

Temporary home of a test suite we are evaluating
Apache License 2.0
2 stars 23 forks source link

U-2-Net_vaiq_int8 model support #188

Open AmosLewis opened 2 months ago

AmosLewis commented 2 months ago

Failer op in U-2-Net_vaiq_int8.default.onnx.torch.elide.mlir

There are 38 resize op failed in this model. The only difference is the inputs[0] tensor size. Here are 2 examples:

%762 = torch.operator "onnx.Resize"(%714, %none, %none, %761) {torch.onnx.coordinate_transformation_mode = "half_pixel", torch.onnx.cubic_coeff_a = -7.500000e-01 : f32, torch.onnx.mode = "linear", torch.onnx.nearest_mode = "floor"} : (!torch.vtensor<[1,32,10,10],f32>, !torch.none, !torch.none, !torch.vtensor<[4],si64>) -> !torch.vtensor<[?,?,?,?],f32>

%5094 = torch.operator "onnx.Resize"(%5046, %none, %none, %5093) {torch.onnx.coordinate_transformation_mode = "half_pixel", torch.onnx.cubic_coeff_a = -7.500000e-01 : f32, torch.onnx.mode = "linear", torch.onnx.nearest_mode = "floor"} : (!torch.vtensor<[1,256,40,40],f32>, !torch.none, !torch.none, !torch.vtensor<[4],si64>) -> !torch.vtensor<[?,?,?,?],f32>

AmosLewis commented 3 weeks ago

The 2 U-2 models failed again. https://github.com/nod-ai/e2eshark-reports/blob/main/2024-06-11/onnx_reports/statusreport.md @zjgarvey can you take a look at?

AmosLewis commented 3 weeks ago 
python ./run.py --torchmlirbuild ../../torch-mlir/build --tolerance 0.001 0.001 --cachedir ./huggingface_cache --ireebuild ../../iree-build  -f onnx -g models --mode onnx --report --tests onnx/models/U-2-Net_vaiq_int8  --torchtolinalg  
Starting e2eshark tests. Using 4 processes
Cache Directory: /home/chi/src/SHARK-TestSuite/e2eshark/huggingface_cache
Tolerance for comparing floating point (atol, rtol) = (0.001, 0.001)
Torch MLIR build: /home/chi/src/torch-mlir/build
IREE build: /home/chi/src/iree-build
Test run directory: /home/chi/src/SHARK-TestSuite/e2eshark/test-run
Since --tests or --testsfile was specified, --groups ignored
Framework:onnx mode=onnx backend=llvm-cpu runfrom=model-run runupto=inference
Test list: ['onnx/models/U-2-Net_vaiq_int8']
Test onnx/models/U-2-Net_vaiq_int8 failed [inference]
Generated status report /home/chi/src/SHARK-TestSuite/e2eshark/test-run/statusreport.md
Generated time report /home/chi/src/SHARK-TestSuite/e2eshark/test-run/timereport.md
Generated summary report /home/chi/src/SHARK-TestSuite/e2eshark/test-run/summaryreport.md

Completed run of e2e shark tests
Status report for run: test-run using mode:onnx todtype:default backend:llvm-cpu

| tests                         | model-run   | onnx-import   | torch-mlir   | iree-compile   | inference   |
|:------------------------------|:------------|:--------------|:-------------|:---------------|:------------|
| onnx/models/U-2-Net_vaiq_int8 | passed      | passed        | passed       | passed         | failed      |

Need to bump torch-mlir in iree