search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.78k stars 7.96k forks source link

network_predict_batch : segmentation fault if batchsize > 1 #6846

Open pfeatherstone opened 4 years ago

pfeatherstone commented 4 years ago

network_predict_batch seg faults when batchsize is greater than 1. Stack trace is : image

pfeatherstone commented 4 years ago

I've tracked the segmentation fault to : fill_cpu(l.outputs*l.batch, 0, l.output, 1); in forward_convolutional_layer. signal is thrown when l.batch > 1

pfeatherstone commented 4 years ago

Found it, set_batch_network does not reallocate all the buffers appropriately. I don't think this was the intended design. @AlexeyAB what do you think?

pfeatherstone commented 4 years ago

Ok the fix is to call resize_network afterwards. This reallocates the buffers. You're welcome world.

pfeatherstone commented 4 years ago

Actually, when built with cuda and batchsize > 1, you get error:

CUDA status = cudaDeviceSynchronize() Error: file: ./src/blas_kernels.cu : () : line: 1011 : build time: Oct 19 2020 - 08:15:30 

 CUDA Error: an illegal memory access was encountered
CUDA Error: an illegal memory access was encountered: File exists
yolov3_darknet: ./src/utils.c:325: error: Assertion `0' failed.
Aborted (core dumped)
pfeatherstone commented 4 years ago

@AlexeyAB any thoughts of this. It would be really great if batch prediction worked on both CPU and GPU

lucheng07082221 commented 4 years ago

@pfeatherstone I want to know this error fixed?

pfeatherstone commented 4 years ago

Nope

pfeatherstone commented 3 years ago

@AlexeyAB Have you had time to look at this?

enesozi commented 3 years ago

Have you tried putting the following code snippet if (net->batch != 1) set_batch_network(net, batch_size); before the line, i.e. network_predict(*net, im.data);

ghost commented 3 years ago

Was this fixed? Can we do batch inference now?

AlexeyAB commented 3 years ago

You can use batch-inference in such a way: https://github.com/AlexeyAB/darknet/pull/7915/files#diff-3db380348fb2b87a1bf487f798535627a2f992109c16a8dc55012e2da75b1af4R380

Using such code: https://github.com/Smorodov/Multitarget-tracker/blob/master/src/Detector/YoloDarknetDetector.cpp#L120-L122

pfeatherstone commented 3 years ago

@AlexeyAB so have the bugs mentioned above been fixed ?

sarimmehdi commented 3 years ago

@AlexeyAB can you show us how to do this in Python (calling the c++ functions in Python)

pfeatherstone commented 3 years ago

The issues I had with batch inference was resizing the batch size at runtime. Say if you want a batch of size 8 but the .cfg file has a batch of size 1 and you called set_batch_network() with possibly resize_network(), you would get a segmentation fault when using CUDA. I think on CPU it worked.

Batch inference only seemed to work if you used whatever was specified in the .cfg file. This is annoying if you want to dynamically change the batch size.