pvelesko
commented
8 months ago Open pvelesko opened 8 months ago
pvelesko
commented
8 months ago pvelesko
commented
8 months ago pvelesko
commented
3 months ago @franz Is there a single reason why most of these fail?
franz
commented
3 months ago IIRC there were multiple reasons. The major one was that the Mali does not support FP64 and there is no emulation (AFAIK). Some of the tests have "int" or "unsigned" in the name, but they have shared kernel code for all variants including "double", and because of that, they driver failed to compile the SPIRV. Another reason was that some tests required more memory than the driver could allocate, this is perhaps fixable with some kernel parameter tuning, i haven't tried. Then there are a bunch of memset/memfill tests that fail, i have no idea why, possibly bugs in Mali drivers.
pvelesko
commented
3 months ago The major one was that the Mali does not support FP64 and there is no emulation (AFAIK). Some of the tests have "int" or "unsigned" in the name, but they have shared kernel code for all variants including "double",
This we can filter out now with -DCHIP_SKIP_DOUBLE_TESTS=ON
IIRC only a couple of tests failed due to this.
pvelesko
commented
2 weeks ago Updated list of failing tests:
Unit_hipGraphAddEventRecordNode_Functional_Simple: ''
Unit_hipGraphAddEventRecordNode_Functional_WithoutFlags: ''
Unit_hipGraphAddEventRecordNode_Functional_WithFlags: ''
Unit_hipGraphAddEventRecordNode_Functional_TimingDisabled: ''
Unit_hipGraphAddEventWaitNode_Functional_Simple: ''
Unit_hipGraphAddEventWaitNode_MultGraphMultStrmDependency: ''
Unit_hipGraphAddEventWaitNode_MultGraphOneStrmDependency: ''
Unit_hipGraphAddEventWaitNode_differentFlags: ''
Unit_hipGraphNodeGetType_NodeType: ''
Unit_hipGraphEventWaitNodeSetEvent_SetProp: ''
Unit_hipStreamBeginCapture_ColligatedStrmCapture_defaultflag: ''
Unit_hipStreamBeginCapture_ColligatedStrmCapture_blockingflag: ''
Unit_hipStreamBeginCapture_ColligatedStrmCapture_diffflags: ''
Unit_hipStreamBeginCapture_ColligatedStrmCapture_diffprio: ''
Unit_hipMemcpy2DToArrayAsync_PinnedHostMemSameGpu: ''
Unit_hipMemcpy_HalfMemCopy: ''
Unit_hipMemcpy_MultiThread-AllAPIs: ''
Unit_hipMemcpyWithStream_TestwithTwoStream: ''
Unit_hipMemcpyWithStream_TestDtoDonSameDevice: ''
Unit_hipMemcpyWithStream_MultiThread: ''
Unit_hipHostMalloc_NonCoherent: ''
Unit_hipHostMalloc_Default: ''
Unit_hipStreamDestroy_WithFinishedWork: ''
Unit_hipStreamCreate_MultistreamBasicFunctionalities: ''
Unit_hipEvent: ''
Unit_hipEventElapsedTime: ''
Unit_hipEventMGpuMThreads_1: ''
Unit_hipDeviceSynchronize_Functional: ''
Unit_hipStreamPerThread_EventSynchronize: ''
Unit_hipMultiThreadStreams1_AsyncSync: ''
Unit_hipMultiThreadStreams1_AsyncAsync: ''
hipMemset_Unit_hipMemsetAsync_SetMemoryWithOffset_Helgrind: ''
TestRecordEventBlocking: ''
MatrixMultiply: ''
hipEvent: ''
shuffle: ''
broadcast: ''
broadcast2: ''
2d_shuffle: ''
unroll: ''
hip_async_binomial: ''
BinomialOption: ''
BitonicSort: ''
DCT: ''
dwtHaar1D: ''
FastWalshTransform: ''
FloydWarshall: ''
Histogram: ''
RecursiveGaussian: ''
PrintfSimple: ''
PrintfNOP: ''
PrintfDynamic: ''
shuffles: ''
graphMatrixMultiply: ''
cuda-asyncAPI: ''
cuda-matrixMul: ''
cuda-bandwidthTest: '' Will work on root-causing these in for 1.3
pvelesko
commented
2 weeks ago The following tests FAILED:
422 - Unit_hipGraphAddEventRecordNode_Functional_Simple (Timeout)
423 - Unit_hipGraphAddEventRecordNode_Functional_WithoutFlags (Timeout)
425 - Unit_hipGraphAddEventRecordNode_Functional_WithFlags (Timeout)
427 - Unit_hipGraphAddEventRecordNode_Functional_TimingDisabled (Timeout)
429 - Unit_hipGraphAddEventWaitNode_Functional_Simple (Timeout)
430 - Unit_hipGraphAddEventWaitNode_MultGraphMultStrmDependency (Timeout)
432 - Unit_hipGraphAddEventWaitNode_MultGraphOneStrmDependency (Timeout)
433 - Unit_hipGraphAddEventWaitNode_differentFlags (Timeout)
492 - Unit_hipGraphNodeGetType_NodeType (Timeout)
510 - Unit_hipGraphEventWaitNodeSetEvent_SetProp (Timeout)
536 - Unit_hipStreamBeginCapture_ColligatedStrmCapture_defaultflag (Timeout)
537 - Unit_hipStreamBeginCapture_ColligatedStrmCapture_blockingflag (Timeout)
538 - Unit_hipStreamBeginCapture_ColligatedStrmCapture_diffflags (Timeout)
539 - Unit_hipStreamBeginCapture_ColligatedStrmCapture_diffprio (Timeout)
630 - Unit_hipMemcpy2DToArrayAsync_PinnedHostMemSameGpu (Timeout)
717 - Unit_hipMemcpy_HalfMemCopy (Timeout)
718 - Unit_hipMemcpy_MultiThread-AllAPIs (Timeout)
831 - Unit_hipMemcpyWithStream_TestwithTwoStream (Timeout)
838 - Unit_hipMemcpyWithStream_TestDtoDonSameDevice (Timeout)
839 - Unit_hipMemcpyWithStream_MultiThread (Timeout)
858 - Unit_hipHostMalloc_NonCoherent (Timeout)
860 - Unit_hipHostMalloc_Default (Timeout)
1043 - Unit_hipStreamDestroy_WithFinishedWork (Timeout)
1044 - Unit_hipStreamCreate_MultistreamBasicFunctionalities (Timeout)
1055 - Unit_hipEvent (Timeout)
1059 - Unit_hipEventElapsedTime (Timeout)
1067 - Unit_hipEventMGpuMThreads_1 (Timeout)
1094 - Unit_hipDeviceSynchronize_Functional (Timeout)
1189 - Unit_hipStreamPerThread_EventSynchronize (Timeout)
1200 - Unit_hipMultiThreadStreams1_AsyncSync (Timeout)
1201 - Unit_hipMultiThreadStreams1_AsyncAsync (Timeout)
1224 - hipMemset_Unit_hipMemsetAsync_SetMemoryWithOffset_Helgrind (Failed)
1284 - TestRecordEventBlocking (Timeout)
1329 - MatrixMultiply (Timeout)
1330 - hipEvent (Timeout)
1333 - shuffle (Failed)
1334 - broadcast (Failed)
1335 - broadcast2 (Failed)
1336 - 2d_shuffle (Failed)
1341 - unroll (Failed)
1354 - hip_async_binomial (Timeout)
1355 - BinomialOption (Timeout)
1356 - BitonicSort (Timeout)
1357 - DCT (Timeout)
1358 - dwtHaar1D (Timeout)
1359 - FastWalshTransform (Timeout)
1360 - FloydWarshall (Timeout)
1361 - Histogram (Timeout)
1362 - RecursiveGaussian (Timeout)
1364 - PrintfSimple (Failed)
1365 - PrintfNOP (Failed)
1366 - PrintfDynamic (Failed)
1367 - shuffles (Failed)
1369 - graphMatrixMultiply (Timeout)
1374 - cuda-asyncAPI (Timeout)
1376 - cuda-matrixMul (Timeout)
1385 - cuda-bandwidthTest (Timeout)For non-timeout failures:
pvelesko@salami:~/actions-runner/_work/chipStar/chipStar/build$ ctest -R "shuffle$|broadcast$|broadcast2$|2d_shuffle$|unroll$|PrintfSimple$|PrintfNOP$|PrintfDynamic$|shuffles$" -V
UpdateCTestConfiguration from :/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/DartConfiguration.tcl
UpdateCTestConfiguration from :/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/DartConfiguration.tcl
Test project /home/pvelesko/actions-runner/_work/chipStar/chipStar/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1333
Start 1333: shuffle
1333: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/4_shfl/shfl"
1333: Test timeout computed to be: 10000000
1333: CHIP warning [TID 158391] [1723457021.633496613] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1333: Device name Mali-G52 r0p0
1333: CHIP error [TID 158391] [1723457021.634605877] : hipErrorNotInitialized (CL_INVALID_VALUE ) in /home/pvelesko/actions-runner/_work/chipStar/chipStar/src/backend/OpenCL/CHIPBackendOpenCL.cc:817:compileIL
1333:
1333: CHIP error [TID 158391] [1723457021.634775635] : Caught Error: hipErrorNotInitialized
1333: ERROR on line 101: 3
1/9 Test #1333: shuffle ..........................***Failed Required regular expression not found. Regex=[PASSED
] 0.58 sec
test 1334
Start 1334: broadcast
1334: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/4_shfl/broadcast"
1334: Test timeout computed to be: 10000000
1334: CHIP warning [TID 158403] [1723457022.211940958] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1334: CHIP error [TID 158403] [1723457022.212910965] : hipErrorNotInitialized (CL_INVALID_VALUE ) in /home/pvelesko/actions-runner/_work/chipStar/chipStar/src/backend/OpenCL/CHIPBackendOpenCL.cc:817:compileIL
1334:
1334: CHIP error [TID 158403] [1723457022.213082016] : Caught Error: hipErrorNotInitialized
1334: ERROR on line 62: 3
2/9 Test #1334: broadcast ........................***Failed Required regular expression not found. Regex=[PASSED
] 0.58 sec
test 1335
Start 1335: broadcast2
1335: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/4_shfl/broadcast2"
1335: Test timeout computed to be: 10000000
1335: CHIP warning [TID 158415] [1723457022.787183852] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1335: CHIP error [TID 158415] [1723457022.788151484] : hipErrorNotInitialized (CL_INVALID_VALUE ) in /home/pvelesko/actions-runner/_work/chipStar/chipStar/src/backend/OpenCL/CHIPBackendOpenCL.cc:817:compileIL
1335:
1335: CHIP error [TID 158415] [1723457022.788319450] : Caught Error: hipErrorNotInitialized
1335: ERROR on line 62: 3
3/9 Test #1335: broadcast2 .......................***Failed Required regular expression not found. Regex=[PASSED
] 0.58 sec
test 1336
Start 1336: 2d_shuffle
1336: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/5_2dshfl/2dshfl"
1336: Test timeout computed to be: 10000000
1336: CHIP warning [TID 158427] [1723457023.382947028] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1336: Device name Mali-G52 r0p0
1336: CHIP error [TID 158427] [1723457023.384035833] : hipErrorNotInitialized (CL_INVALID_VALUE ) in /home/pvelesko/actions-runner/_work/chipStar/chipStar/src/backend/OpenCL/CHIPBackendOpenCL.cc:817:compileIL
1336:
1336: CHIP error [TID 158427] [1723457023.384205800] : Caught Error: hipErrorNotInitialized
1336: ITEM 0 OK
1336: ITEM: 1 cpu: 40 gpu: 0
1336: ITEM: 2 cpu: 80 gpu: 2.8026e-45
1336: ITEM: 3 cpu: 120 gpu: 0
1336: ITEM: 4 cpu: 10 gpu: 2.8026e-45
1336: ITEM: 5 cpu: 50 gpu: 0
1336: ITEM: 6 cpu: 90 gpu: 2.8026e-45
1336: ITEM: 7 cpu: 130 gpu: 0
1336: ITEM: 8 cpu: 20 gpu: 2.8026e-45
1336: ITEM: 9 cpu: 60 gpu: 0
1336: ITEM: 10 cpu: 100 gpu: 2.8026e-45
1336: ITEM: 11 cpu: 140 gpu: 0
1336: ITEM: 12 cpu: 30 gpu: 2.8026e-45
1336: ITEM: 13 cpu: 70 gpu: 0
1336: ITEM: 14 cpu: 110 gpu: 2.8026e-45
1336: ITEM: 15 cpu: 150 gpu: 0
1336: FAIL: 15 errors
4/9 Test #1336: 2d_shuffle .......................***Failed Required regular expression not found. Regex=[PASSED
] 0.64 sec
test 1341
Start 1341: unroll
1341: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/9_unroll/unroll"
1341: Test timeout computed to be: 10000000
1341: CHIP warning [TID 158439] [1723457024.016589235] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1341: Device name Mali-G52 r0p0
1341: CHIP error [TID 158439] [1723457024.017691374] : hipErrorNotInitialized (CL_INVALID_VALUE ) in /home/pvelesko/actions-runner/_work/chipStar/chipStar/src/backend/OpenCL/CHIPBackendOpenCL.cc:817:compileIL
1341:
1341: CHIP error [TID 158439] [1723457024.017855174] : Caught Error: hipErrorNotInitialized
1341: 1 cpu: 40.000000 gpu 0.000000
1341: 2 cpu: 80.000000 gpu 0.000000
1341: 3 cpu: 120.000000 gpu 0.000000
1341: 4 cpu: 10.000000 gpu 0.000000
1341: 5 cpu: 50.000000 gpu 0.000000
1341: 6 cpu: 90.000000 gpu 0.000000
1341: 7 cpu: 130.000000 gpu 0.000000
1341: 8 cpu: 20.000000 gpu 0.000000
1341: 9 cpu: 60.000000 gpu 0.000000
1341: 10 cpu: 100.000000 gpu 0.000000
1341: 11 cpu: 140.000000 gpu 0.000000
1341: 12 cpu: 30.000000 gpu 0.000000
1341: 13 cpu: 70.000000 gpu 0.000000
1341: 14 cpu: 110.000000 gpu 0.000000
1341: 15 cpu: 150.000000 gpu 0.000000
1341: FAILED: 15 errors
5/9 Test #1341: unroll ...........................***Failed Required regular expression not found. Regex=[PASSED
] 0.59 sec
test 1364
Start 1364: PrintfSimple
1364: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/printf/strings"
1364: Test timeout computed to be: 10000000
1364: CHIP warning [TID 158451] [1723457024.603341126] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1364: PASSED!
6/9 Test #1364: PrintfSimple .....................***Failed Required regular expression not found. Regex=[## no_args.*Hello.*## literal_str_arg.*Hello strings.*## global_str_arg.*HELLO.* STRiNGS.*PASSED!
] 2.47 sec
test 1365
Start 1365: PrintfNOP
1365: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/printf/nop_printfs"
1365: Test timeout computed to be: 10000000
1365: CHIP warning [TID 158463] [1723457027.072899529] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1365: PASSED!
7/9 Test #1365: PrintfNOP ........................***Failed Required regular expression not found. Regex=[Howdy HIPpies!.*I'm correct since I'm not an empty string!.*PASSED!
] 1.26 sec
test 1366
Start 1366: PrintfDynamic
1366: Test command: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/bin/spirv-extractor "--check-for-doubles" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/printf/dynamic_str_args"
1366: Test timeout computed to be: 10000000
1366: CHIP warning [TID 158475] [1723457028.347460986] : The device might not support subgroup size 32, warp-size sensitive kernels might not work correctly.
1366: PASSED!
8/9 Test #1366: PrintfDynamic ....................***Failed Required regular expression not found. Regex=[## conditional string.*inout\[0\] was 1 1.*## array of strings, selecting index 1.*I am a dynamic str arg B.*## host defined string: Extremely dynamic printf str!.*## host defined string with skip: mely dynamic printf str!.*PASSED!
] 2.53 sec
test 1367
Start 1367: shuffles
1367: Test command: /usr/bin/cmake "-P" "/home/pvelesko/actions-runner/_work/chipStar/chipStar/build/samples/shuffles/run-shuffles.cmake"
1367: Test timeout computed to be: 10000000
1367: CMake Error at run-shuffles.cmake:19 (message):
1367: FAIL: Standard output does not match 'shuffles.xstdout'
1367:
1367:
9/9 Test #1367: shuffles .........................***Failed 0.68 sec
0% tests passed, 9 tests failed out of 9
Label Time Summary:
internal = 9.23 sec*proc (8 tests)
Total Test time (real) = 10.08 sec
The following tests FAILED:
1333 - shuffle (Failed)
1334 - broadcast (Failed)
1335 - broadcast2 (Failed)
1336 - 2d_shuffle (Failed)
1341 - unroll (Failed)
1364 - PrintfSimple (Failed)
1365 - PrintfNOP (Failed)
1366 - PrintfDynamic (Failed)
1367 - shuffles (Failed)
Errors while running CTest
Output from these tests are in: /home/pvelesko/actions-runner/_work/chipStar/chipStar/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.Conclusions:
CL_INVALID_VALUE, I extracted SPIR-V from one of these and my guess would be it's because we have this in the SPIR-V: OpSubgroupShuffleINTEL
pjaaskel
commented
2 weeks ago Does it support the KHR subgroup shuffles? Did you notice in devicelib.cl:
// Use the Intel versions for now by default, since the Intel OpenCL CPU
// driver still implements only them, not the KHR versions.
#define sub_group_shuffle intel_sub_group_shuffle
#define sub_group_shuffle_xor intel_sub_group_shuffle_xorIf it supports the KHR one, we could detect at build time that we are building on non-Intel and use that by the default. Ideally, this would be selected at .spv linkage time in chipStar.
pvelesko
commented
2 weeks ago cl_khr_subgroups 0x400000 (1.0.0)
cl_khr_subgroup_extended_types 0x400000 (1.0.0)
cl_khr_subgroup_non_uniform_vote 0x400000 (1.0.0)
cl_khr_subgroup_ballot 0x400000 (1.0.0)
cl_khr_subgroup_non_uniform_arithmetic 0x400000 (1.0.0)
cl_khr_subgroup_shuffle 0x400000 (1.0.0)
cl_khr_subgroup_shuffle_relative 0x400000 (1.0.0)
cl_khr_subgroup_clustered_reduce 0x400000 (1.0.0)Seems like it is supported.
pvelesko
commented
2 weeks ago So seems like this is one of the issues but there are more - I haven't investigated every single SPIR-V.