[A770] Issues with the tutorials

pbchekin commented 6 months ago

Umbrella issue to track Triton tutorials issues with A770.

10-experimental-block-pointer with TRITON_INTEL_ENABLE_BLOCK_PTR=1 fails:

malloc(): invalid size (unsorted)
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libtriton.so         0x0000789c71bb96d7
1  libtriton.so         0x0000789c71bb715c
2  libtriton.so         0x0000789c71bb9daf
3  libc.so.6            0x0000789d6df6f520
4  libc.so.6            0x0000789d6dfc39fc pthread_kill + 300
5  libc.so.6            0x0000789d6df6f476 raise + 22
6  libc.so.6            0x0000789d6df557f3 abort + 211
7  libc.so.6            0x0000789d6dfb6676
8  libc.so.6            0x0000789d6dfcdcfc
9  libc.so.6            0x0000789d6dfd10dc
10 libc.so.6            0x0000789d6dfd2139 malloc + 153
11 libstdc++.so.6       0x0000789d5a54d98c operator new(unsigned long) + 28
12 libigc.so.1          0x0000789c[433](https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/9507665230/job/26207534324#step:20:434)edf3c std::vector<unsigned char, std::allocator<unsigned char>>::_M_default_append(unsigned long) + 380
13 libigc.so.1          0x0000789c439ddc58
14 libigc.so.1          0x0000789c439de0b8
15 libigc.so.1          0x0000789c439de302
16 libigc.so.1          0x0000789c439ff23a
17 libigc.so.1          0x0000789c439ff9d6
18 libigc.so.1          0x0000789c43a00d04
19 libigc.so.1          0x0000789c43a03ad8
20 libigc.so.1          0x0000789c4390fd45
21 libigc.so.1          0x0000789c438dacff
22 libigc.so.1          0x0000789c438e4187
23 libigc.so.1          0x0000789c438e45da
24 libigc.so.1          0x0000789c43748693
25 libigc.so.1          0x0000789c43844b82
26 libigc.so.1          0x0000789c[436](https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/9507665230/job/26207534324#step:20:437)2b322
27 libigc.so.1          0x0000789c43382605
28 libigc.so.1          0x0000789c43efe4de llvm::FPPassManager::runOnFunction(llvm::Function&) + 702
29 libigc.so.1          0x0000789c43efe804 llvm::FPPassManager::runOnModule(llvm::Module&) + 52
30 libigc.so.1          0x0000789c43eff59c llvm::legacy::PassManagerImpl::run(llvm::Module&) + 812
31 libigc.so.1          0x0000789c433fd2e2
32 libigc.so.1          0x0000789c43094aba
33 libigc.so.1          0x0000789c432e575b
34 libigc.so.1          0x0000789c43097587
35 libigc.so.1          0x0000789c4310a5ed
36 libigc.so.1          0x0000789c4310c3bc
37 libze_intel_gpu.so.1 0x0000789c66561eac
38 libze_intel_gpu.so.1 0x0000789c665630bc
39 libze_intel_gpu.so.1 0x0000789c6693e2e7
40 libze_intel_gpu.so.1 0x0000789c664beb4f
41 spirv_utils.so       0x0000789d5068db38 create_module(_ze_context_handle_t*, _ze_device_handle_t*, unsigned int*, unsigned long) + 88
42 spirv_utils.so       0x0000789d5068ea2c
43 libpython3.9.so.1.0  0x0000789d6e2c7fc8
44 libpython3.9.so.1.0  0x0000789d6e2a5d90 _PyObject_MakeTpCall + 128
45 libpython3.9.so.1.0  0x0000789d6e2ff375 _PyEval_EvalFrameDefault + 21077
46 libpython3.9.so.1.0  0x0000789d6e2a66d3
47 libpython3.9.so.1.0  0x0000789d6e2a7f41
48 libpython3.9.so.1.0  0x0000789d6e2feee2 _PyEval_EvalFrameDefault + 19906
49 libpython3.9.so.1.0  0x0000789d6e2f95b8
50 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
51 libpython3.9.so.1.0  0x0000789d6e2d8c08
52 libpython3.9.so.1.0  0x0000789d6e201bdc
53 libpython3.9.so.1.0  0x0000789d6e2fa8c7 _PyEval_EvalFrameDefault + 1959
54 libpython3.9.so.1.0  0x0000789d6e2f95b8
55 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
56 libpython3.9.so.1.0  0x0000789d6e2a7f41
57 libpython3.9.so.1.0  0x0000789d6e2a726a PyVectorcall_Call + 202
58 libpython3.9.so.1.0  0x0000789d6e2fdb74 _PyEval_EvalFrameDefault + 14932
59 libpython3.9.so.1.0  0x0000789d6e2f95b8
60 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
61 libpython3.9.so.1.0  0x0000789d6e2fa514 _PyEval_EvalFrameDefault + 1012
62 libpython3.9.so.1.0  0x0000789d6e2f95b8
63 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
64 libpython3.9.so.1.0  0x0000789d6e2fb375 _PyEval_EvalFrameDefault + [469](https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/9507665230/job/26207534324#step:20:470)3
65 libpython3.9.so.1.0  0x0000789d6e2f95b8
66 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
67 libpython3.9.so.1.0  0x0000789d6e2a7f41
68 libpython3.9.so.1.0  0x0000789d6e2a726a PyVectorcall_Call + 202
69 libpython3.9.so.1.0  0x0000789d6e2fdb74 _PyEval_EvalFrameDefault + 14932
70 libpython3.9.so.1.0  0x0000789d6e2f95b8
71 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
72 libpython3.9.so.1.0  0x0000789d6e2fa514 _PyEval_EvalFrameDefault + 1012
73 libpython3.9.so.1.0  0x0000789d6e2f95b8
74 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
75 libpython3.9.so.1.0  0x0000789d6e2a7f41
76 libpython3.9.so.1.0  0x0000789d6e2a726a PyVectorcall_Call + 202
77 libpython3.9.so.1.0  0x0000789d6e2fdb74 _PyEval_EvalFrameDefault + 14932
78 libpython3.9.so.1.0  0x0000789d6e2f95b8
79 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
80 libpython3.9.so.1.0  0x0000789d6e2fb375 _PyEval_EvalFrameDefault + 4693
81 libpython3.9.so.1.0  0x0000789d6e2f95b8
82 libpython3.9.so.1.0  0x0000789d6e2a6505 _PyFunction_Vectorcall + 229
83 libpython3.9.so.1.0  0x0000789d6e2fa514 _PyEval_EvalFrameDefault + 1012
84 libpython3.9.so.1.0  0x0000789d6e2f95b8
85 libpython3.9.so.1.0  0x0000789d6e2f92e5 _PyEval_EvalCodeWithName + 85
86 libpython3.9.so.1.0  0x0000789d6e2f9277 PyEval_EvalCodeEx + 71
87 libpython3.9.so.1.0  0x0000789d6e37278f PyEval_EvalCode + 31
88 libpython3.9.so.1.0  0x0000789d6e384fc4
89 libpython3.9.so.1.0  0x0000789d6e384f4b
90 libpython3.9.so.1.0  0x0000789d6e24aa48
91 libpython3.9.so.1.0  0x0000789d6e24a7d8 PyRun_SimpleFileExFlags + 911
92 libpython3.9.so.1.0  0x0000789d6e38d641 Py_RunMain + 769
93 libpython3.9.so.1.0  0x0000789d6e38d1cd Py_BytesMain + 61
94 libc.so.6            0x0000789d6df56d90
95 libc.so.6            0x0000789d6df56e40 __libc_start_main + 128
96 python3              0x00005d1801a[490](https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/9507665230/job/26207534324#step:20:491)95 _start + 37

pbchekin commented 5 months ago

A new run with Agama 881.19 and PTDB 0.5.1:

02-fused-softmax now works.
06-fused-attention now works.
10-experimental-block-pointer test (renamed from 09-experimental-block-pointer) with TRITON_INTEL_ENABLE_BLOCK_PTR=1 still fails.

https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/9507665230/job/26207534324

anmyachev commented 5 months ago

Hi @whitneywhtsang, should we remove tutorial 10-experimental-block-pointer.py as it is already removed in triton itself https://github.com/triton-lang/triton/pull/3371/files?

whitneywhtsang commented 5 months ago

Hi @whitneywhtsang, should we remove tutorial 10-experimental-block-pointer.py as it is already removed in triton itself https://github.com/triton-lang/triton/pull/3371/files?

We intentionally kept it, as we are using the tutorial to measure our GEMM performance.

alexbaden commented 4 months ago

With #1316 we should be able to add this to the tutorial skiplist for A770.

intel / intel-xpu-backend-for-triton

[A770] Issues with the tutorials #1081