[tracking] HF CNN fp32 model tests

[jinchen62]

Below is the list of HF CNN fp32 model issues for Full model list. Tests are imported in https://github.com/nod-ai/SHARK-TestSuite/pull/313.

Status on 9/12:	items	tests	model-run	onnx-import	torch-mlir	iree-compile	inference
total-count	529	499	489	418	310	154

Steps: Build torch-mlir and iree locally

cd SHARK-TestSuite/e2eshark/
python ./run.py  --tolerance 0.001 0.001 -c path/to/torch-mlir/build -i path/to/iree-build --cachedir path/to/cache -f onnx -g models --mode onnx --report -j 12 --torchtolinalg --testsfile model_list.txt

model-run

• [ ] ONNXRuntimeError (30 models)

  onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/optimizer/initializer.cc:49 onnxruntime::Initializer::Initializer(const onnx::TensorProto&, const onnxruntime::Path&) [ONNXRuntimeError] : 1 : FAIL : GetFileLength for ./patch_embed.proj.weight failed:Invalid fd was supplied: -1

onnx-import

• [ ] opset version issue (10 models)
• [x] miss shape inference (303 models) https://github.com/llvm/torch-mlir/issues/3593 Note: Setting opset vertsion to 21

torch-mlir

• [ ] onnx.Expand with empty shape result (47 models)
• [ ] onnx.If with different result shapes in then & else branches (2 models) https://github.com/nod-ai/SHARK-Turbine/issues/696
• [ ] onnx.LSTM with bidirectional direction attr (1 model) https://github.com/nod-ai/SHARK-Turbine/issues/315
• [ ] onnx.Mul with shape mismatch (3 models)
• [ ] onnx.Resize with cubic mode attr (11 models)
• [ ] crash (7 models)
• [x] torch.aten.squeeze (113 models) https://github.com/llvm/torch-mlir/pull/3675
• [x] onnx.BatchNormalization (61 models) https://github.com/llvm/torch-mlir/pull/3597
• [x] onnx.AveragePool (5 models)
• [x] onnx.Conv (10 models)
• [x] onnx.Split crash (6 models)
• [x] onnx.Transpose (280 models)
• [x] killed (8 models)

iree-compile

• [ ] 'util.initializer' op failed to inline into combined initializer (14 models) https://github.com/iree-org/iree/issues/18386
• [ ] ~~'func.func' op exceeded stack allocation limit (3 models) https://github.com/iree-org/iree/issues/18187~~
• [ ] 'stream.async.dispatch' op has invalid Read access range (1 model)
• [ ] 'linalg.conv_2d_nchw_fchw' op inferred input/output operand dim mismatch (65 models) https://github.com/iree-org/iree/issues/18387
• [ ] 'linalg.generic' op inferred input/output operand dim mismatch (9 models)
• [ ] 'vector.shape_cast' op source/result number of elements must match (16 models)
• [ ] expected offsets to be non-negative, but got -1 (1 model)
• [ ] crash (1 model)
• [x] jit global source initialization order invalid: global "__hoisted_i32" has no value (90 models) https://github.com/iree-org/iree/issues/18232
• [x] integer value too large (40 models) https://github.com/llvm/torch-mlir/issues/3621 Note: Use --torchtolinalg flag to lower the IR to linalg through torch-mlir

inference

• [ ] mismatch (152 models)
• [ ] failed (1 model)

[AmosLewis]

onnx.Conv in #566 has been fixed by disable quantized 1d conv. This should be a different conv error.

[AmosLewis]

Better to add the failed op into https://github.com/nod-ai/SHARK-Turbine/issues/797

[AmosLewis]

onnx.Batchnorm in https://github.com/nod-ai/SHARK-Turbine/issues/566 also has been fixed by bump iree. This should be a different error. Maybe you should check vivek's patch https://github.com/llvm/torch-mlir/pull/3597

[vivekkhandelwal1]

Hi @jinchen62, can you please update this issue, since it seems some of these have been fixed.

[jinchen62]

@vivekkhandelwal1 Nothing was changed by Monday, will verify and update weekly.