FPS slow down

[AndrewSivrit]

Hi @ganyc717 I find one more problem. After run darknet_cl.exe detector demo data/voc.data yolo-voc.cfg yolo-voc.weights test.mp4 On the first frames I get 7,1 FPS, but after several minutes I get less 5 FPS.

[AndrewSivrit]

If run to train speed of learning is slow down too

[Kylin-PHYTIUM]

The performance of CL is lower than corresponding CUDA implementation on the same device. @ganyc717 @AndrewSivrit I want to known that whether the code supports train yolo2 on multiple CL devices?

[ganyc717]

Hi @Kylin-PHYTIUM I have checked the bottleneck, the performance blocked at BLAS library, I think nvidia has their own well optimization cuBLAS library which is about 3 times faster than clBLAS. Currently multi device is not supported, maybe I will add support later.

[Bardo91]

The same issue happens to me. When I let it run for a while, the speed starts decreasing. Here I plot the times in microseconds over time for some of the required stages. All of them remain stable (img preparation, data copy, result copying etc...) but the prediction grows continuously.

To be exact, the call that grows in time is the call to the prediction of the net from the data, i.e. network_predict(net, X);.

Any clue?

[Bardo91]

Just to isolate the problem I observed what happens if I reload the "network *net" object each time I use it. Basically I call net = load_network(wStr1, wStr2, 0); at the beginning of the loop and free_network(net); at the end. It can be clearly seen in the following figure that the "slow down" does not happens if the network is reloaded:

Left figure shows that if the net is not reloaded, i.e. it is loaded just at the beginning (which is the logical behaviour), then the prediction time increases. However, right figure shows that if the net is reloaded each time, then no slow down can be observed, but obviously, loading the network takes a couple of seconds, so it is not a solution... I will keep digging into the problem, I hope this helps to someone else too.

[Bardo91]

I am not an expert in neither YOLO nor CL, but while debugging a bit the code I found that the code that slows down over time is https://github.com/ganyc717/Darknet-On-OpenCL/blob/master/darknet_cl/src/network.cpp#L781. Particularly, using tiny_yolo_v2, the layer that causes the issue is the layer number 15, which seems to be the region_layer.... The deeper I went was this line https://github.com/ganyc717/Darknet-On-OpenCL/blob/c13fefc66a13da5805986937fccd486b2b313c24/darknet_cl/src/blas_kernels_cl.cpp#L1020. If I am not wrong, that line queue the call of the kernel, what can cause that continuous increment of time?

[Bardo91]

I modified the code to enable CL_QUEUE_PROFILING_ENABLE. Then I added the following lines to profile the enqueue call:

cl_ulong time_start;
cl_ulong time_end;
clGetEventProfilingInfo(e, CL_PROFILING_COMMAND_START, sizeof(time_start), &time_start, NULL);
clGetEventProfilingInfo(e, CL_PROFILING_COMMAND_END, sizeof(time_end), &time_end, NULL);
double nanoSeconds = time_end-time_start;
printf("OpenCl Execution time is: %0.3f milliseconds \n",nanoSeconds / 1000000.0);

These measures of time remain stable... which confuses me more. It seems that the GPU run it self takes the same time, but when the measure of the cpu call it grows in time:

clock_t t1 = clock();
cl_event e;
cl_int status = clEnqueueNDRangeKernel(*cl->queue, kernel, 3, NULL, global_size, NULL, NULL, NULL, &e);
clock_t t2 = clock();
printf("enqueue : \t %f\n",(float)(t2 - t1) / CLOCKS_PER_SEC);