nod-ai / SHARK-ModelDev

Unified compiler/runtime for interfacing with PyTorch Dynamo.
Apache License 2.0
95 stars 48 forks source link

[Tracker] All the issue related with ONNX model zoo models #886

Open pdhirajkumarprasad opened 1 week ago

pdhirajkumarprasad commented 1 week ago

We have enabled total 2325 models from ONNX model zoo. This issue is to track all the issues seen in the models.

Passing Summary

TOTAL TESTS = 2325 Stage # Passing % of Total % of Attempted
Setup 2319 99.7.0% 99.7.0%
IREE Compilation 1910 82.1% 82.1%
Gold Inference 1910 82.1% 82.3%
IREE Inference Invocation 1825 78.4% 95.5%
Inference Comparison (PASS) 1268 54.5% 69.4%

Fail Summary

TOTAL TESTS = 2325 Stage # Failed at Stage % of Total
Setup 6 0.2%
IREE Compilation 409 17.5%
Gold Inference 0 0.0%
IREE Inference Invocation 95 4.0%
Inference Comparison 557 23.9%

CPU

# issue type issue no #model impacted list of model assignee status
1 failed to legalize operation 'arith.sitofp' that was explicitly marked illegal 151
2 failed to legalize operation 'torch.operator' that was explicitly marked illegal 56
3 'tensor.expand_shape' op expected dimension 1 of collapsed type to be static value of 9216 29
4 'linalg.fill' op unexpected result less than 0 at expression #1 in (d0, d1, d2, d3, d4) -> (d0, d1, d2, d3, d4) 6
5 'linalg.generic' op inferred input/output operand #2 has shape's dimension #3 to be 392, but found 4 14
6 One or more operations with large vector sizes (16384 bytes) were found 27
7 expected '>' 34
8 'func.func' op exceeded stack allocation limit of 32768 bytes for function. Got 33792 bytes 4
9 'vector.transfer_write' op inferred mask type ('vector<1x1x8xi1>') and mask operand type ('vector<1x8x1xi1>') don't match 9
10 failed to legalize operation 'torch.aten.nonzero' 8
11 expected type to be 'tensor<?x?x?xf32>' or a rank-reduced version. (size mismatch) 3
12 'tensor.cast' op operand type 'tensor<80x15130xf32>' and result type 'tensor<15130xf32>' are cast incompatibl 3
13 'tensor.reshape' op source and destination tensor should have the same number of elements 56