Closed inferrna closed 7 years ago
You have to add support for some debug parameters/env-vars (if still doesn't). For example, if I had access to failed kernel source I would try to compile it with CodeXL and compose more reliable bugreport.
Yes, alright, good idea. What is your preferred interface for this? (eg, create an env var? Preferred var name etc?)
(thinking about this though, segfault is probably not during opencl build. bugs in kernel build will cause the sourcecode to be dumped into /tmp/failed-kernel.cl This is already the case)
(though, I dont know, so let's try getting the opencl sourcecode dumped, from env var, and we can see)
For example we could have:
Note that in the general case there might be a bunch of different kernels/sourcecodes being dumped. So we might want to include the kernel name, or the sourcefile name, in that somehow/somewhere.
COCL_DUMP_KERNEL=/tmp/foo.cl
- good variant, at least we can get source of the last failed kernel.
I thinking about passing directory for loading fixed kernels, like this
COCL_LOAD_KERNEL_FROM=/tmp/kernels/
- where /tmp/kernels/ can contain kernels and other functions in separate files. For example, if code find /tmp/kernels/function_or_kernel_name.cl - it would overwrite own function/kernel variant with its content.
k, sounds good. What I'm going to do:
libcocl.so
as an so, means I can hack around with libcocl.so
without having to rebuild entire tensorflow wheel...libcocl.so
, to do add the debuggin options abovelibcocl.so
inside it with a new one
libcocl.so
is at: (your virtualenv)/lib/python3.5/site-packages/tensorflow/third_party/cuda-on-cl/libcocl.so
(I mean, on your machine; I havent actually create the modified version for download yet)
(Note that if you decide to try modifying libcocl.so in the meantime, you'll need the latest version, from latest cuda-on-cl master
branch, and make sure you rebuild evrything in cuda-on-cl, such that the following commands all show some output:objdump -x -a build/libcocl.so | grep RPATH
objdump -x -a build/libeasycl.so | grep RPATH
objdump -x -a build/clblast/libclblast.so | grep RPATH
)
I'd prefer to build it from source. Maybe it is a good point to start to use cmake? There is at least 2 parameters need to be configured: DEBUG=ON/OFF USE_INTERNAL_CLBLAST=ON/OFF (I use clblast as a separate package for caffe)
I'd prefer to build it from source.
Ok. If you're up from buliding from source, you could hack the file direclty perhaps. Basically, in src/hostside_opencl_funcs.cpp, scroll down to the method getKernelForName
, which does:
kernelByname
mapCLKernel *kernel = cl->buildKernelFromString(sourcecode, name, "", "__internal__");
, you can paste the following code block: cout << "load env " << (void *)getenv("LOAD") << endl;
bool load = getenv("LOAD") != 0;
string filename = "/tmp/out.cl";
if(load) {
cout << "loading kernel" << endl;
ifstream f;
f.open(filename, ios_base::in);
// f << launchConfiguration.kernelName << endl;
// f >> sourcecode;
sourcecode = "";
string line = "";
while(getline(f, line)) {
sourcecode += line + "\n";
}
// cout << sourcecode << endl;
f.close();
} else {
cout << "saving kernel" << endl;
ofstream f;
f.open(filename, ios_base::out);
// f << launchConfiguration.kernelName << endl;
f << sourcecode << endl;
f.close();
}
(I coded this before we discussed above; I was already using this code, just was setting the bool load
variable by hand. It sort of kind of works, and then we can kind of bash it into doing what you want. Currently:
/tmp/out.cl
,LOAD=1
, in whichc case it will load the sourcecode from this file instead, and compile it, run itMaybe it is a good point to start to use cmake?
I'm up for using cmake. I like cmake. The only gentle concern I have with cmake is that it makes it hard to build using anything other than gcc
, on linux. And currently everything ecxept linking is using clang
. But thinking it through:
libcocl.so
, ir-to-opencl
and patch-hostside
, it probably doesnt matter if they're built with gcc or clang. I'm not quite sure :-) We could trycocl.Makefile
definitely must be built using clang
, .. .but thats a different makefile, different than the Makefile
, in the root of cuda-on-cl source repository.So... I think that we probably can try migrating the top-level Makefile
to cmake, and see what happens.
What are your thougths on who will migrate to CMakeLists.txt? I can take a look if you want?
Well... I coudlnt put up with manky Makefile
dependencies any more, and added a CMaekLists.txt in https://github.com/hughperkins/cuda-on-cl/commit/22c16dac81bde2f74e22f80c7679c708c5a3833b
(CMakeLists.txt Looking pretty clean in latest version; though latest version breaks tensorflow build, so ... :-P )
File missed or wrong declaration:
CMake Error at CMakeLists.txt:115 (add_library):
Cannot find source file:
src/CLBlast/src/database/database.cpp
Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
.hxx .in .txx
CMake Error: CMake can not determine linker language for target: clblast
CMake Error: Cannot determine link language for target "clblast".
CMake Error in CMakeLists.txt:
Exporting the target "clblast" is not allowed since its linker language
cannot be determined
I have manually pulled CLBlast and other submodules before doing cmake.
Hi inferrna,
Can you provide the output of git submodule
please?
$ git submodule
c09e59fae751bfa912886aea6000f41b301665a0 src/CLBlast (c09e59f)
5dee35e0853d6cfc57f921586dd96968a9e602c3 src/EasyCL (v1.3-246-g5dee35e)
Also, can you provide hte output of git status
pelase?
git status
$ git submodule
+d190becd89d4747d2cfd5d77e821d4e84ad01941 ../src/CLBlast (0.6.0)
+2fbae1f9e12ca4d85079bfba7a3a0fffed03ec7d ../src/EasyCL (v1.3-249-g2fbae1f)
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: ../CMakeLists.txt
difference (with my cmake changes - just added add_subdirectory(src/CLBlast)
and removed some stuff)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 10f1d0f..cc80550 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -89,30 +89,31 @@ set(CLBLAST_ROUTINES ${CLBLAST_LEVEL1_ROUTINES} ${CLBLAST_LEVEL2_ROUTINES} ${CLB
set(CLBLAST_PRECISIONS 32 64 3232 6464 16)
# Gathers all source-files
-set(CLBLAST_SOURCES
- src/CLBlast/src/database/database.cpp
- src/CLBlast/src/routines/common.cpp
- src/CLBlast/src/cache.cpp
- src/CLBlast/src/clblast.cpp
- src/CLBlast/src/clblast_c.cpp
- src/CLBlast/src/routine.cpp
- src/CLBlast/src/utilities.cpp
-)
-foreach(ROUTINE ${CLBLAST_LEVEL1_ROUTINES})
- set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/level1/${ROUTINE}.cpp)
-endforeach()
-foreach(ROUTINE ${CLBLAST_LEVEL2_ROUTINES})
- set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/level2/${ROUTINE}.cpp)
-endforeach()
-foreach(ROUTINE ${CLBLAST_LEVEL3_ROUTINES})
- set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/level3/${ROUTINE}.cpp)
-endforeach()
-foreach(ROUTINE ${CLBLAST_LEVELX_ROUTINES})
- set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/levelx/${ROUTINE}.cpp)
-endforeach()
-
-include_directories(src/CLBlast/src)
-add_library(clblast SHARED ${CLBLAST_SOURCES})
+add_subdirectory(src/CLBlast)
+#set(CLBLAST_SOURCES
+# src/CLBlast/src/database.cc
+# src/CLBlast/src/routines/common.cpp
+# src/CLBlast/src/cache.cpp
+# src/CLBlast/src/clblast.cpp
+# src/CLBlast/src/clblast_c.cpp
+# src/CLBlast/src/routine.cpp
+# src/CLBlast/src/utilities.cpp
+#)
+#foreach(ROUTINE ${CLBLAST_LEVEL1_ROUTINES})
+# set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/level1/${ROUTINE}.cpp)
+#endforeach()
+#foreach(ROUTINE ${CLBLAST_LEVEL2_ROUTINES})
+# set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/level2/${ROUTINE}.cpp)
+#endforeach()
+#foreach(ROUTINE ${CLBLAST_LEVEL3_ROUTINES})
+# set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/level3/${ROUTINE}.cpp)
+#endforeach()
+#foreach(ROUTINE ${CLBLAST_LEVELX_ROUTINES})
+# set(CLBLAST_SOURCES ${CLBLAST_SOURCES} src/CLBlast/src/routines/levelx/${ROUTINE}.cpp)
+#endforeach()
+#
+#include_directories(src/CLBlast/src)
+#add_library(clblast SHARED ${CLBLAST_SOURCES})
target_include_directories(clblast PRIVATE src/CLBlast/include)
@@ -156,7 +157,7 @@ INSTALL(FILES ${CLEW_HEADERS} DESTINATION include)
INSTALL(FILES ${EASYCL_HEADERS_ROOT} DESTINATION include/EasyCL)
# INSTALL(FILES ${CMAKE_SOURCE_DIR}/cmake/cocl.cmake DESTINATION share/cocl)
INSTALL(FILES ${CMAKE_BINARY_DIR}/cmake/cocl.cmake DESTINATION share/cocl)
-install(TARGETS easycl clew clblast cocl ir-to-opencl patch-hostside EXPORT cocl-targets
+install(TARGETS easycl clew cocl ir-to-opencl patch-hostside EXPORT cocl-targets
LIBRARY DESTINATION lib
ARCHIVE DESTINATION lib
RUNTIME DESTINATION bin
diff --git a/src/CLBlast b/src/CLBlast
index c09e59f..d190bec 160000
--- a/src/CLBlast
+++ b/src/CLBlast
@@ -1 +1 @@
-Subproject commit c09e59fae751bfa912886aea6000f41b301665a0
+Subproject commit d190becd89d4747d2cfd5d77e821d4e84ad01941
diff --git a/src/EasyCL b/src/EasyCL
index 5dee35e..2fbae1f 160000
--- a/src/EasyCL
+++ b/src/EasyCL
@@ -1 +1 @@
-Subproject commit 5dee35e0853d6cfc57f921586dd96968a9e602c3
+Subproject commit 2fbae1f9e12ca4d85079bfba7a3a0fffed03ec7d
Also cmake has own technique to use submodules - just for example. https://coderwall.com/p/y3zzbq/use-cmake-enabled-libraries-in-your-cmake-project
Also cmake has own technique to use submodules - just for example. https://coderwall.com/p/y3zzbq/use-cmake-enabled-libraries-in-your-cmake-project
it does. And I use it for example in DeepCL https://github.com/hughperkins/DeepCL/blob/master/CMakeLists.txt#L66-L74
Kind of fiddly to get working though. Much easier to just build it, without going the whole cmake submodules route... as you can see :-)
Question: what is your use-case that you are trying to solve by replacing the existing clblast build in the cuda-on-cl cmakelists.txt makefile?
After execute
tensorflow-cl$ git submodule update --init --recursive
trying to update cuda-on-cl
tensorflow-cl/third_party/cuda-on-cl$ git pull
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.
git pull <remote> <branch>
tensorflow-cl/third_party/cuda-on-cl$ git status
HEAD detached at d6c2c7a
there seems like `git submodule update --init --recursive' switches all submodules to concrete commit each time. To fix this I executed
git submodule foreach --recursive git checkout master
git submodule foreach --recursive git pull
in tensorflow-cl/third_party/cuda-on-cl and tensorflow-cl. And after I just fixed CMakeLists.txt to make it working with updated CLBlasts own CMakeLists.txt because it sources were much changed.
Ok. Because you want the latest version of CLBlast? I dont support that for now though. You will need to figure out how to get that working on your own :-)
Closing since much of this is done, and much of the rest is out of date. Lets create some new issues for the latest v0.13.0 code.
You have to add support for some debug parameters/env-vars (if still doesn't). For example, if I had access to failed kernel source I would try to compile it with CodeXL and compose more reliable bugreport.