Closed psyhtest closed 7 years ago
Oh, interesting. Yes I noted such errors before. Do you already know what piece of code tries to allocate size 0 buffers?
Do you already know what piece of code tries to allocate size 0 buffers?
No, I was hoping you would have a better idea :).
Please take a look at the sample trace when you have time.
Hm alright, I suspect it might be a stupid code added in the early days, possibly greentea.cpp
.
That'll have to be cleaned up anyways.
https://github.com/naibaf7/caffe/blob/master/src/caffe/greentea/greentea.cpp
So my current hypothesis is that this "workaround" is there for places where it is OK to pass nullptr
into the CUDA kernel, but we can't pass a nullptr
-cl_mem
object into the kernel (CL_INVALID_MEM_OBJECT), so the workaround was to allocate a temporary buffer of size 0. But apparently some OpenCL SDKs are also not OK with that and will fail either early (on allocation) or later on.
As I see it the problem exists only with the pooling layers, and I'll look for another workaround now.
Should hopefully be resolved with the latest commits (+ a few other bugfixes). Tested runtests on AMD and nVidia Linux OpenCL implementation.
@naibaf7
On the GTX 1080, the same benchmark now runs with errcode=0
for all the OpenCL API calls. It still segfaults though, but I believe it's unrelated:
I0209 11:41:03.430191 47885 caffe.cpp:475] Total Time: 144.459 ms.
I0209 11:41:03.430194 47885 caffe.cpp:476] *** Benchmark ends ***
*** Aborted at 1486640463 (unix time) try "date -d @1486640463" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x1100) received by PID 47885 (TID 0x7f45d4895ac0) from PID 4352; stack trace: ***
@ 0x7f45d2a1c4b0 (unknown)
@ 0x7f45d3d22f72 caffe::SyncedMemory::~SyncedMemory()
@ 0x7f45d3d13b02 boost::detail::sp_counted_impl_p<>::dispose()
@ 0x7f45d3cc191a boost::detail::sp_counted_base::release()
@ 0x7f45d3ccf542 boost::detail::sp_counted_impl_p<>::dispose()
@ 0x7f45d3cad4b9 boost::detail::sp_counted_impl_p<>::dispose()
@ 0x7f45d3caa6a8 std::vector<>::~vector()
@ 0x7f45d2a2136a __cxa_finalize
@ 0x7f45d3ca4573 (unknown)
I'll test on the Mali-T628 next... Just need to remember to build using:
$ ck install package:lib-caffe-bvlc-opencl-viennacl-universal \
--env.DISABLE_DOUBLE_SUPPORT=ON \
--env.DISABLE_DEVICE_HOST_UNIFIED_MEMORY=ON \
--env.CK_HOST_CPU_NUMBER_OF_PROCESSORS=2
I've checked on the Odroid XU3 platform, and this problem has gone away. I'm struggling with another issue now, but I'll open a new ticket for that. Thanks.
I'm investigating some failures apparently caused by OpenCL API calls to create zero-sized buffers when benchmarking with
caffe time
. I've seen such failures for a while, but it's only now I am beginning to get some sense of what's going on with help of dividiti's OpenCL profiler.For example, on an Ubuntu 16.04 workstation with the NVIDIA GTX 1080 GPU, the benchmark completes with:
On a development platform with the ARM Mali-T628 GPU, the benchmark may not even complete with:
along the way.
Such calls seem to happen across BLAS libraries and models, but to get the ball rolling let's consider ViennaCL with AlexNet.
The easiest way to reproduce is via CK-Caffe:
Install CK:
Install CK-Caffe:
Install Caffe with ViennaCL:
Run Caffe benchmarking under dividiti's OpenCL profiler:
Inspect the profiler's output:
The output is a list of dictionaries describing traced OpenCL API calls. At 5-6% of the total trace, there's a call to
clCreateBuffer()
withsize=0
, which returnserrcode=-61
(CL_INVALID_BUFFER_SIZE
) as per the specification.On the Mali-powered platform things go downhill from there causing failures in
clEnqueueNDRangeKernel()
and even segfaults inclEnqueueWriteBuffer()
...Please find attached a sample CK-Caffe session and profiler trace from the GTX 1080 powered workstation:
tmp-dvdt-prof.json
above)