Slow FPS on Tesla P100

[ghost]

Hello,

I am getting slow fps (~24) while using a Tesla p100 GPU on a 480x270 video. I have played around with configurations but cannot find a way to increase it. Testing the same video using darknet on a GTX 1080 is getting around 60 fps.

Below are the details of my setup:

GPU:

Makefile:

GPU=1
CUDNN=1
CUDNN_HALF=0
OPENCV=1
AVX=0
OPENMP=1
LIBSO=0
ZED_CAMERA=0
#set GPU=1 and CUDNN=1 to speedup on GPU
 # set CUDNN_HALF=1 to further speedup 3 x times (Mixed-precision on Tensor Cores) GPU: Volta, Xavier, Turing and higher
 # set AVX=1 and OPENMP=1 to speedup on CPU (if error occurs then set AVX=0)

 DEBUG=0

# ARCH= -gencode arch=compute_30,code=sm_30 \
   #    -gencode arch=compute_35,code=sm_35 \
   #    -gencode arch=compute_50,code=[sm_50,compute_50] \
   #    -gencode arch=compute_52,code=[sm_52,compute_52] \
   #    -gencode arch=compute_61,code=[sm_61,compute_61]

OS := $(shell uname)

                      # Tesla V100
# ARCH= -gencode arch=compute_70,code=[sm_70,compute_70]

 # GeForce RTX 2080 Ti, RTX 2080, RTX 2070, Quadro RTX 8000, Quadro RTX 6000, Quadro RTX 5000, Tesla T4, XNOR Tensor Cores
 # ARCH= -gencode arch=compute_75,code=[sm_75,compute_75]

 # Jetson XAVIER
 # ARCH= -gencode arch=compute_72,code=[sm_72,compute_72]

 # GTX 1080, GTX 1070, GTX 1060, GTX 1050, GTX 1030, Titan Xp, Tesla P40, Tesla P4
 # ARCH= -gencode arch=compute_61,code=sm_61 -gencode arch=compute_61,code=compute_61

 # GP100/Tesla P100 - DGX-1
ARCH= -gencode arch=compute_60,code=sm_60

 # For Jetson TX1, Tegra X1, DRIVE CX, DRIVE PX - uncomment:
 # ARCH= -gencode arch=compute_53,code=[sm_53,compute_53]

 # For Jetson Tx2 or Drive-PX2 uncomment:
 # ARCH= -gencode arch=compute_62,code=[sm_62,compute_62]

VPATH=./src/
EXEC=darknet
OBJDIR=./obj/

ifeq ($(LIBSO), 1)
LIBNAMESO=libdarknet.so
APPNAMESO=uselib
endif

CC=gcc
CPP=g++
NVCC=/usr/local/cuda/bin/nvcc
OPTS=-Ofast
LDFLAGS= -lm -pthread
COMMON= -Iinclude/ -I3rdparty/stb/include
CFLAGS=-Wall -Wfatal-errors -Wno-unused-result -Wno-unknown-pragmas -fPIC

ifeq ($(DEBUG), 1)
#OPTS= -O0 -g
#OPTS= -Og -g
COMMON+= -DDEBUG
CFLAGS+= -DDEBUG
else
ifeq ($(AVX), 1)
CFLAGS+= -ffp-contract=fast -mavx -mavx2 -msse3 -msse4.1 -msse4.2 -msse4a
endif
endif

CFLAGS+=$(OPTS)

ifeq ($(OPENCV), 1)
COMMON+= -DOPENCV
CFLAGS+= -DOPENCV
LDFLAGS+= `pkg-config --libs opencv`
COMMON+= `pkg-config --cflags opencv`
endif

ifeq ($(OPENMP), 1)
CFLAGS+= -fopenmp
LDFLAGS+= -lgomp
endif

ifeq ($(GPU), 1)
COMMON+= -DGPU -I/usr/local/cuda/include/
CFLAGS+= -DGPU
ifeq ($(OS),Darwin) #MAC
LDFLAGS+= -L/usr/local/cuda/lib -lcuda -lcudart -lcublas -lcurand
else
LDFLAGS+= -L/usr/local/cuda/lib64 -lcuda -lcudart -lcublas -lcurand
endif
endif

ifeq ($(CUDNN), 1)
COMMON+= -DCUDNN
ifeq ($(OS),Darwin) #MAC
CFLAGS+= -DCUDNN -I/usr/local/cuda/include
LDFLAGS+= -L/usr/local/cuda/lib -lcudnn
else
CFLAGS+= -DCUDNN -I/usr/local/cudnn/include
LDFLAGS+= -L/usr/local/cudnn/lib64 -lcudnn
endif
endif

ifeq ($(CUDNN_HALF), 1)
COMMON+= -DCUDNN_HALF
CFLAGS+= -DCUDNN_HALF
ARCH+= -gencode arch=compute_70,code=[sm_70,compute_70]
endif

ifeq ($(ZED_CAMERA), 1)
CFLAGS+= -DZED_STEREO -I/usr/local/zed/include
LDFLAGS+= -L/usr/local/zed/lib -lsl_core -lsl_input -lsl_zed
#-lstdc++ -D_GLIBCXX_USE_CXX11_ABI=0 
endif

OBJ=image_opencv.o http_stream.o gemm.o utils.o dark_cuda.o convolutional_layer.o list.o image.o activations.o im2col.o col2im.o blas.o crop_layer.o dropout_layer.o maxpool_layer.o softmax_layer.o data.o matrix.o network.o connected_layer.o cost_layer.o parser.o option_list.o darknet.o detection_layer.o captcha.o route_layer.o writing.o box.o nightmare.o normalization_layer.o avgpool_layer.o coco.o dice.o yolo.o detector.o layer.o compare.o classifier.o local_layer.o swag.o shortcut_layer.o activation_layer.o rnn_layer.o gru_layer.o rnn.o rnn_vid.o crnn_layer.o demo.o tag.o cifar.o go.o batchnorm_layer.o art.o region_layer.o reorg_layer.o reorg_old_layer.o super.o voxel.o tree.o yolo_layer.o upsample_layer.o lstm_layer.o
ifeq ($(GPU), 1) 
LDFLAGS+= -lstdc++ 
OBJ+=convolutional_kernels.o activation_kernels.o im2col_kernels.o col2im_kernels.o blas_kernels.o crop_layer_kernels.o dropout_layer_kernels.o maxpool_layer_kernels.o network_kernels.o avgpool_layer_kernels.o
endif

OBJS = $(addprefix $(OBJDIR), $(OBJ))
DEPS = $(wildcard src/*.h) Makefile include/darknet.h

all: obj backup results setchmod $(EXEC) $(LIBNAMESO) $(APPNAMESO)

ifeq ($(LIBSO), 1)
CFLAGS+= -fPIC

$(LIBNAMESO): $(OBJS) include/yolo_v2_class.hpp src/yolo_v2_class.cpp
    $(CPP) -shared -std=c++11 -fvisibility=hidden -DLIB_EXPORTS $(COMMON) $(CFLAGS) $(OBJS) src/yolo_v2_class.cpp -o $@ $(LDFLAGS)

$(APPNAMESO): $(LIBNAMESO) include/yolo_v2_class.hpp src/yolo_console_dll.cpp
        $(CPP) -std=c++11 $(COMMON) $(CFLAGS) -o $@ src/yolo_console_dll.cpp $(LDFLAGS) -L ./ -l:$(LIBNAMESO)
endif

$(EXEC): $(OBJS)
        $(CPP) -std=c++11 $(COMMON) $(CFLAGS) $^ -o $@ $(LDFLAGS)
$(OBJDIR)%.o: %.c $(DEPS)
        $(CC) $(COMMON) $(CFLAGS) -c $< -o $@

$(OBJDIR)%.o: %.cpp $(DEPS)
        $(CPP) -std=c++11 $(COMMON) $(CFLAGS) -c $< -o $@

$(OBJDIR)%.o: %.cu $(DEPS)
        $(NVCC) $(ARCH) $(COMMON) --compiler-options "$(CFLAGS)" -c $< -o $@

obj:
    mkdir -p obj
backup:
    mkdir -p backup
results:
    mkdir -p results
setchmod:
    chmod +x *.sh

.PHONY: clean
clean:
        rm -rf $(OBJS) $(EXEC) $(LIBNAMESO) $(APPNAMESO)

How could I go about speeding up the fps? I need it to be around the 60 mark for an application I am working on. I have also used ( cmake . && make ) to install darknet and this did not improve the fps. Thank you!

[AlexeyAB]

@jksLearning Hi,

• Show screenshot of output of this command nvcc --version

• What command do you use for detection?

• Did you get this screenshot during Darknet execution?

GPU:

[ghost]

1.

2. ./darknet detector demo cfg/coco.data cfg/yolov3.cfg yolov3.weights /home/jshede/timelicameramodule/images/videos/DMTV01/2019-04-22/2019-04-22-19-46-31.mp4 -don't_show

3. No that command was not run during execution. Here is during execution.

[AlexeyAB]

• Do you use the latest version of Darknet?
• Did you change anything in cfg-file?
• What CUDA, cuDNN and OpenCV versions do you use?
• What is the CPU-usage during detection on video? top command
• Try to detect on another video-file

Try to keep un-commented these lines: https://github.com/AlexeyAB/darknet/blob/cce34712f6928495f1fbc5d69332162fc23491b9/Makefile#L16-L20 And comment this line: https://github.com/AlexeyAB/darknet/blob/cce34712f6928495f1fbc5d69332162fc23491b9/Makefile#L37

If it doesn't help, try to update your GPU-drivers.

[ghost]

1. Yes, I cloned the repo today and did not get any improvement

2. Nothing was changed in cfg-file

3. • CUDA: 9.0
   • CuDNN: 7.1.1
   • OpenCV: 3.4.0

4. CPU usage ranges from 10%-25%

5. Tested on another video and got 23 fps

6. Same result with lines 16-20 uncommented and line 37 commented out.

I appreciate you helping out with this!

[AlexeyAB]

• Can you show screenshot of output of top command?

• Try to compile with LIBSO=1 in the Makefile and run this command, what FPS can you get?

LD_LIBRARY_PATH=./:$LD_LIBRARY_PATH ./uselib data/coco.names cfg/yolov3.cfg yolov3.weights test.mp4

[ghost]

Top output while running ./darknet detector demo cfg/coco.data cfg/yolov3.cfg yolov3.weights /home/jshede/timelicameramodule/images/videos/DMTV01/2019-04-22/2019-04-22-19-46-31.mp4 -dont_show

The command you suggested fails because

I use -dont_show with the other command because I did not install libgtk2.0 with OpenCV as I am running on a cluster that requires me to ssh into, Is there a similar option for uselib?

[ghost]

Updated my drivers and am getting this error now. Have you ever encountered this error prior @AlexeyAB ?

CUDA status Error: file: /home/jshede/timelicameramodule/stalled_veh/darknet/src/dark_cuda.c : () : line: 36 : build time: Apr 30 2019 - 20:27:45 CUDA Error: unknown error CUDA Error: unknown error: Bad file descriptor darknet: /home/jshede/timelicameramodule/stalled_veh/darknet/src/utils.c:293: void error(const char*): Assertion0' failed. Aborted (core dumped)`

My output from running make looks the same as before. Not quite sure why this is happening.

[AlexeyAB]

@jksLearning It's a CUDA error. I don't know why.