search

KhronosGroup / webcl-validator

WebCL Validator
Other
38 stars 8 forks source link

regression tests fail #103

Open reuk opened 8 years ago

reuk commented 8 years ago

I followed the build procedure from the readme and some of the regression tests fail.

What does this indicate, and should I be worried? Does it mean my hardware is broken or has bad opencl support, my drivers are buggy, or the validator is buggy?

OS X 10.11.3, 2.5 GHz Intel Core i7, Iris Pro + AMD Radeon R9 M370X

Failing Tests (17):
    WebCL Validator :: access-pointer.cl
    WebCL Validator :: attack.cl
    WebCL Validator :: builtin-vload-indirect.cl
    WebCL Validator :: builtin-vload-offset.cl
    WebCL Validator :: builtin-vload-wrapper.cl
    WebCL Validator :: builtin-vstore-wrapper.cl
    WebCL Validator :: complete_transformation.cl
    WebCL Validator :: include-option.cl
    WebCL Validator :: parallel-merge-local.cl
    WebCL Validator :: private_mem_zeroing.cl
    WebCL Validator :: reading_invalid_addresses.cl
    WebCL Validator :: relocate-variables.cl
    WebCL Validator :: remove-variables.cl
    WebCL Validator :: rename-variables.cl
    WebCL Validator :: structure-as-function-parameter.cl
    WebCL Validator :: write-image.cl
    WebCL Validator :: zero-local-mem.cl

  Expected Passes    : 66
  Unexpected Failures: 17
elhigu commented 8 years ago

Hi, Sorry for delay, I was on vacation last week and had super busy week before that (bad excuse). I have exact same setup here so I'll try this out and report back, which part I suspect causing errors...

elhigu commented 8 years ago

OSX opencl drivers has been always, pretty bad. I tried to disable all drivers except CPU in https://github.com/KhronosGroup/webcl-validator/blob/master/test/include/driver_blacklist.hpp

It did help a bit, then only 4 tests are failing, IIRC those 4 tests were failing also 2 years ago, so no much improvement has happened on OSX opencl CPU driver :( Probably you would have better success trying to run code generated by validator on Windows and with intel drivers. What is your use case for this?

btw. you can run separate test cases from command line by copy-pasting test script of failed test to investigate more closely what went wrong and what part of protected code causes compiler to fail.

like this: 

Mikaels-MacBook-Pro:build mikaelle$ make check-webcl-validator
[  0%] Built target check-empty-memory
...
[100%] Built target radix-sort
[100%] Running WebCL Validator regression tests
FAIL: WebCL Validator :: builtin-vstore-wrapper.cl (11 of 83)
******************** TEST 'WebCL Validator :: builtin-vstore-wrapper.cl' FAILED ********************
Script:
--
sed 's/@SIZE@/4/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator"
sed 's/@SIZE@/4/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" - | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator"
sed 's/@SIZE@/4/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" - | grep -v CHECK | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./FileCheck" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl"
sed 's/@SIZE@/4/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" - | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./kernel-runner" --webcl --kernel builtin_wrappers --global float 7 | grep '^1,2,3,4,-1,-1,-1,0'
sed 's/@SIZE@/4/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" - | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./kernel-runner" --webcl --kernel builtin_wrappers --global float 8 | grep '^1,2,3,4,1,2,3,4,0'
sed 's/@SIZE@/8/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" - | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./kernel-runner" --webcl --kernel builtin_wrappers --global float 7 | grep '^-1,-1,-1,-1,-1,-1,-1,0'
sed 's/@SIZE@/8/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" - | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./kernel-runner" --webcl --kernel builtin_wrappers --global float 8 | grep '^1,2,3,4,5,6,7,8,0'
sed 's/@SIZE@/8/g' "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/builtin-vstore-wrapper.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" - | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./kernel-runner" --webcl --kernel builtin_wrappers --global float 16 | grep '^1,2,3,4,5,6,7,8,1,2,3,4,5,6,7,8,0'
--
Exit Code: 1

Command Output (stdout):
--
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.

--
Command Output (stderr):
--
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Can't build OpenCL program.
CVMS_ERROR_SERVICE_FAILURE: CVMS compiler has crashed or hung managing the service.
--

********************
FAIL: WebCL Validator :: include-option.cl (36 of 83)
******************** TEST 'WebCL Validator :: include-option.cl' FAILED ********************
Script:
--
cat "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include/1st-level.h" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include-option.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator" -I"/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include"
"/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include-option.cl" -I"/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include" -include 1st-level.h | grep -v IMPLICIT | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator"
"/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include-option.cl" -I"/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include" -include 1st-level.h 2>&1 | grep -v CHECK | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./FileCheck" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/include-option.cl"
--
Exit Code: 1

Command Output (stdout):
--
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Iris Pro.
Skipping blacklisted driver
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/AMD Radeon R9 M370X Compute Engine.
Skipping blacklisted driver
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.

--
Command Output (stderr):
--
1 warning and 2 errors generated.
Error while processing /var/tmp//wclwGkEPj.
warning: argument unused during compilation: '-fsyntax-only'
error: WebCL doesn't support the include directive.

#include "2nd-level.h"
error: '2nd-level.h' file not found
#include "2nd-level.h"
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Can't build OpenCL program.
No kernels or only kernel prototypes found when build executable.
--

********************
FAIL: WebCL Validator :: radix-sort/input.cl (48 of 83)
******************** TEST 'WebCL Validator :: radix-sort/input.cl' FAILED ********************
Script:
--
"/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator" < "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/radix-sort/input.cl"
"/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./radix-sort" -original < "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/radix-sort/input.cl"
"/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/radix-sort/input.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./radix-sort" -transform
--
Exit Code: 1

Command Output (stdout):
--
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Iris Pro.
Skipping blacklisted driver
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/AMD Radeon R9 M370X Compute Engine.
Skipping blacklisted driver
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./radix-sort: Compiling input for Apple/AMD Radeon R9 M370X Compute Engine.
Skipping blacklisted driver

--
Command Output (stderr):
--
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./radix-sort: Can't create unsorted element buffer.

--

********************
FAIL: WebCL Validator :: remove-variables.cl (72 of 83)
******************** TEST 'WebCL Validator :: remove-variables.cl' FAILED ********************
Script:
--
"/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator" < "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/remove-variables.cl"
"/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/remove-variables.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator"
--
Exit Code: 1

Command Output (stdout):
--
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Iris Pro.
Skipping blacklisted driver
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/AMD Radeon R9 M370X Compute Engine.
Skipping blacklisted driver
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.

--
Command Output (stderr):
--
1 warning generated.
1 warning generated.
1 warning generated.
1 warning generated.
warning: argument unused during compilation: '-fsyntax-only'
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Can't build OpenCL program.
CVMS_ERROR_COMPILER_FAILURE: CVMS compiler has crashed or hung building an element.
--

********************
Testing Time: 2.22s
********************
Failing Tests (4):
    WebCL Validator :: builtin-vstore-wrapper.cl
    WebCL Validator :: include-option.cl
    WebCL Validator :: radix-sort/input.cl
    WebCL Validator :: remove-variables.cl

  Expected Passes    : 79
  Unexpected Failures: 4
make[3]: *** [tools/clang/tools/webcl-validator/test/CMakeFiles/check-webcl-validator] Error 1
make[2]: *** [tools/clang/tools/webcl-validator/test/CMakeFiles/check-webcl-validator.dir/all] Error 2
make[1]: *** [tools/clang/tools/webcl-validator/test/CMakeFiles/check-webcl-validator.dir/rule] Error 2
make: *** [check-webcl-validator] Error 2
Mikaels-MacBook-Pro:build mikaelle$

So for test number 72:

Mikaels-MacBook-Pro:build mikaelle$ "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator" < "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/remove-variables.cl"; echo $?
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Iris Pro.
Skipping blacklisted driver
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/AMD Radeon R9 M370X Compute Engine.
Skipping blacklisted driver
0
Mikaels-MacBook-Pro:build mikaelle$ "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./webcl-validator" "/Users/mikaelle/Projects/Vincit/webcl/temp/llvm/tools/clang/tools/webcl-validator/test/remove-variables.cl" | "/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator"; echo $?
1 warning generated.
1 warning generated.
1 warning generated.
1 warning generated.
warning: argument unused during compilation: '-fsyntax-only'
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Compiling input for Apple/Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz.
/Users/mikaelle/Projects/Vincit/webcl/temp/build/bin/./opencl-validator: Can't build OpenCL program.
CVMS_ERROR_COMPILER_FAILURE: CVMS compiler has crashed or hung building an element.1
Mikaels-MacBook-Pro:build mikaelle$

You can see that the problem was the second command in test script.