intel / torch-xpu-ops

Apache License 2.0
30 stars 21 forks source link

Case fail (segfault and failures) due to oneDNN matmul #157

Open fengyuan14 opened 7 months ago

fengyuan14 commented 7 months ago

🐛 Describe the bug

A case fail due to oneDNN matmul implementation,

image

Skip the case temporarily.

Versions

Latest pytorch.

fengyuan14 commented 7 months ago

test_dtypes_nn_functional_multi_head_attention_forward_xpu test_dtypes_nn_functional_linear_xpu test_dtypes_pca_lowrank_xpu test_dtypes_svd_lowrank_xpu test_noncontiguous_samples_nn_functional_linear_xpu_int64 test_dtypes__refs_nn_functional_pdist_xpu

285 case failures https://github.com/intel/torch-xpu-ops/pull/158

fengyuan14 commented 7 months ago

https://github.com/intel/torch-xpu-ops/issues/128

daisyden commented 3 weeks ago

On Windows we can also observe fatal issue.

We got access violation with this shape on Windows LNL pytorch UT test_dtypes_nn_functional_linear_xpu.

onednn_verbose,info,oneDNN v3.5.3 (commit 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) onednn_verbose,info,cpu,runtime:threadpool,nthr:8 onednn_verbose,info,cpu,isa:Intel AVX2 with Intel DL Boost onednn_verbose,info,gpu,runtime:DPC++ onednn_verbose,info,gpu,engine,0,backend:Level Zero,name:Intel(R) Arc(TM) 140V GPU (16GB),driver_version:1.3.29550,binary_kernels:enabled

....

onednn_verbose,primitive,exec,gpu,matmul,ocl:gemm_with_po:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 bia_s8::blocked:ab::f0_mask2 dst_s8::blocked:ab::f0,attr-scratchpad:user,,1x3:3x4,0.5107 onednn_verbose,primitive,exec,gpu,matmul,ocl:gemm_with_po:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 bia_s8::blocked:ab::f0_mask2 dst_s8::blocked:ab::f0,attr-scratchpad:user,,8x3:3x4,0.5298 onednn_verbose,primitive,exec,gpu,matmul,ocl:gemm_with_po:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 bia_s8::blocked:ab::f0_mask2 dst_s8::blocked:ab::f0,attr-scratchpad:user,,6x3:3x4,0.2379 onednn_verbose,primitive,exec,gpu,matmul,ocl:gemm_with_po:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 bia_s8::blocked:ab::f0_mask2 dst_s8::blocked:ab::f0,attr-scratchpad:user,,1x8:8x8,0.2293 onednn_verbose,primitive,exec,gpu,matmul,ocl:gemm_with_po:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 bia_s8::blocked:ab::f0_mask2 dst_s8::blocked:ab::f0,attr-scratchpad:user,,8x8:8x8,0.2342 onednn_verbose,primitive,exec,gpu,matmul,ocl:gemm_with_po:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 bia_s8::blocked:ab::f0_mask2 dst_s8::blocked:ab::f0,attr-scratchpad:user,,6x8:8x8,0.2247 onednn_verbose,primitive,exec,gpu,matmul,jit:gemm:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 dst_s8::blocked:ab::f0,attr-scratchpad:user,,1x3:3x4,0.4556 onednn_verbose,primitive,exec,gpu,matmul,jit:gemm:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 dst_s8::blocked:ab::f0,attr-scratchpad:user,,8x3:3x4,0.3466 onednn_verbose,primitive,exec,gpu,matmul,jit:gemm:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 dst_s8::blocked:ab::f0,attr-scratchpad:user,,6x3:3x4,0.1223 onednn_verbose,primitive,exec,gpu,matmul,jit:gemm:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 dst_s8::blocked:ab::f0,attr-scratchpad:user,,1x8:8x8,0.0732 onednn_verbose,primitive,exec,gpu,matmul,jit:gemm:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 dst_s8::blocked:ab::f0,attr-scratchpad:user,,8x8:8x8,0.0603 onednn_verbose,primitive,exec,gpu,matmul,jit:gemm:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 dst_s8::blocked:ab::f0,attr-scratchpad:user,,6x8:8x8,0.0592 onednn_verbose,primitive,exec,gpu,matmul,jit:gemm:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 dst_s8::blocked:ab::f0,attr-scratchpad:user,,4x2:2x4,0.0557 onednn_verbose,primitive,exec,gpu,matmul,ocl:gemm_with_po:any,undef,src_s8::blocked:ab::f0 wei_s8::blocked:ba::f0 bia_s8::blocked:ab::f0_mask2 dst_s8::blocked:ab::f0,attr-scratchpad:user,,4x2:2x4,0.9499 Windows fatal exception: access violation

Thread 0x00000ab4 (most recent call first):

git clone -b pytorch-2.5 https://github.com/intel-innersource/frameworks.ai.pytorch.private-gpu http://mengfeil-ubuntu.sh.intel.com/pytorch/xpu/chuanqiw/intel-deep-learning-essentials-2025.0.0.310_offline.exe pip install http://mengfeil-ubuntu.sh.intel.com/pytorch/xpu/chuanqiw/torch-2.5.0a0%2Bgite84e33f-cp310-cp310-win_amd64.whl