search

imsoo / fight_detection

Real time Fight Detection Based on 2D Pose Estimation and RNN Action Recognition
MIT License
193 stars 42 forks source link

Client ERROR #7

Open wtnan2003 opened 4 years ago

wtnan2003 commented 4 years ago

Server and client make successfully

But Something gets wrong when I run ./darknet_client: image

Is that OpenCV version problem? any help would be appreciated

wtnan2003 commented 4 years ago
I change main.cpp as you described in this Line From To
142 CAP_PROP_FPS CV_CAP_PROP_FPS
143 CAP_PROP_BUFFERSIZE CV_CAP_PROP_BUFFERSIZE
158 CAP_PROP_FPS CV_CAP_PROP_FPS
159 CAP_PROP_FRAME_COUNT CV_CAP_PROP_FRAME_COUNT
184 VideoWriter::fourcc('M', 'P', '4', 'V') CV_FOURCC('M', 'P', '4', 'V')
imsoo commented 4 years ago

@wtnan2003

I think cv::VideoWriter in OpenCV 2.4 doesn't support mp4v format.

Try below changes.

Line From To
184 CV_FOURCC('M', 'P', '4', 'V') CV_FOURCC('X', 'V', 'I', 'D') OR CV_FOURCC('M', 'P', '2', 'V') 
/* 
  opencv/modules/highgui/src/cap_gstreamer.cpp#L462-L479 
*/
void CvVideoWriter_GStreamer::init()
{
    encs[CV_FOURCC('D','R','A','C')]=(char*)"diracenc";
    encs[CV_FOURCC('H','F','Y','U')]=(char*)"ffenc_huffyuv";
    encs[CV_FOURCC('J','P','E','G')]=(char*)"jpegenc";
    encs[CV_FOURCC('M','J','P','G')]=(char*)"jpegenc";
    encs[CV_FOURCC('M','P','1','V')]=(char*)"mpeg2enc";
    encs[CV_FOURCC('M','P','2','V')]=(char*)"mpeg2enc";
    encs[CV_FOURCC('T','H','E','O')]=(char*)"theoraenc";
    encs[CV_FOURCC('V','P','8','0')]=(char*)"vp8enc";
    encs[CV_FOURCC('H','2','6','4')]=(char*)"x264enc";
    encs[CV_FOURCC('X','2','6','4')]=(char*)"x264enc";
    encs[CV_FOURCC('X','V','I','D')]=(char*)"xvidenc";
    encs[CV_FOURCC('F','F','Y','U')]=(char*)"y4menc";
    //encs[CV_FOURCC('H','F','Y','U')]=(char*)"y4menc";
    pipeline=0;
    buffer=0;
}
wtnan2003 commented 4 years ago

@imsoo thanks for giving respond I change to Opencv 3.1 solve it, but still gets some problem:

I have open four terminals and run: ./sink ./ventilator ./worker cfg/fight.cfg weight/fight.weight -gpu 0 -pose ./darknet_client -addr 127.0.0.1 -vid kick.mp4 -out_vid -dont_show

sink work like this: image

ventilator: image

worker : image

darknet_client, keep running all the time : image

cilent

the output video and json is empty : image

what should I do to fix it?

imsoo commented 4 years ago

@wtnan2003

In this repo, we need to make below pipeline.

Client → Ventilator → Worker → (Sink ⇄ Action) → Client

So, you need to run action.py additionally. Try below command.

./sink
./ventilator
./worker cfg/openpose.cfg weight/openpose.weight -gpu 0 -pose
./action.py
./darknet_client ...

If you want to run this(fight detection based on YOLO object detection), try below command using darknet_server not this repo.

./sink
./ventilator
./worker cfg/fight.cfg weights/fight.weights names/fight.names -gpu 0 -thresh 0.2
./darknet_client ...

If this is confusing, let me know and I will try to explain.

wtnan2003 commented 4 years ago

@imsoo thanks for your reply!

I want to run this: image

There are not "openpose.weight" in weight file , only action.h5

wtnan2003 commented 4 years ago

I see I will download it in https://drive.google.com/file/d/1BfY0Hx2d2nm3I4JFh0W1cK2aHD1FSGea/view

wtnan2003 commented 4 years ago

when I run python3 action.py image

my keras version is 2.2.4 tensorflow.keras version is 2.1.6-tf tensorflow version is 1.9.0

what should I do？

thx

wtnan2003 commented 4 years ago

@imsoo Would you tell me your Keras and TensorFlow version?

I am really interested in this repo

imsoo commented 4 years ago

@wtnan2003

Sorry for my late reply. I was late finding the cause of the problem.

First, I've tested the same environment on my computer and got a same error.

import tensorflow as tf
print(tf.__version__)
>>> 1.9.0

from tensorflow import keras
print(keras.__version__)
>>> 2.1.6-tf

I've found this issue. So I've upgrade tensorflow to 2.1.0 and retry it. and it works fine.

import tensorflow as tf
print(tf.__version__)
>>> 2.1.0

from tensorflow import keras
print(keras.__version__)
>>> 2.2.4-tf

Please upgrade tensorflow and try again.

wtnan2003 commented 4 years ago

@imsoo TensorFlow 2.1.0 seems to be incompatible with my system environment: Cuda 9.0.

I worried about the global environment if I change to Cuda 10.1

Is there any easier way to fix it?

imsoo commented 4 years ago

@wtnan2003

I understand it well.

Could you try changed code below?

This code create a standard model with the same architecture and load weights files.

Changed code (action.py) ```python3 import tensorflow as tf import numpy as np import zmq import io import time from tensorflow import keras from tensorflow.keras.backend import set_session config = tf.ConfigProto() config.gpu_options.allow_growth = True sess = tf.Session(config=config) set_session(sess) sess.run(tf.global_variables_initializer()) ''' ## TF 2.0 from tensorflow.compat.v1.keras.backend import set_session tf.compat.v1.disable_eager_execution() config = tf.compat.v1.ConfigProto() config.gpu_options.allow_growth = True sess = tf.compat.v1.Session(config=config) set_session(sess) sess.run(tf.compat.v1.global_variables_initializer()) ''' n_input = 24 # num input parameters per timestep n_steps = 32 n_hidden = 34 # Hidden layer num of features n_classes = 4 batch_size = 1024 from tensorflow.keras import layers lambda_loss_amount = 0.0015 model = tf.keras.Sequential([ # relu activation layers.Dense(n_hidden, activation='relu', kernel_initializer='random_normal', bias_initializer='random_normal', batch_input_shape=(batch_size, n_steps, n_input) ), # cuDNN layers.CuDNNLSTM(n_hidden, return_sequences=True, unit_forget_bias=1.0), layers.CuDNNLSTM(n_hidden, unit_forget_bias=1.0), # layers.LSTM(n_hidden, return_sequences=True, unit_forget_bias=1.0), # layers.LSTM(n_hidden, unit_forget_bias=1.0), layers.Dense(n_classes, kernel_initializer='random_normal', bias_initializer='random_normal', kernel_regularizer=tf.keras.regularizers.l2(lambda_loss_amount), bias_regularizer=tf.keras.regularizers.l2(lambda_loss_amount), activation='softmax' ) ]) model.load_weights("weights/action.h5") def load_X(msg): buf = io.StringIO(msg) X_ = np.array( [elem for elem in [ row.split(',') for row in buf ]], dtype=np.float32 ) blocks = int(len(X_) / 32) X_ = np.array(np.split(X_,blocks)) return X_ # load input_ = np.zeros((batch_size, n_steps, n_input), dtype=np.float32) print("model loaded ...") context = zmq.Context() socket = context.socket(zmq.REP) socket.bind("ipc://action") while True: msg = socket.recv() msg = msg.decode("utf-8") recv_ = load_X(msg) for i in range(len(recv_)): input_[i] = recv_[i] startTime = time.time() pred = model.predict_classes(input_, batch_size = batch_size) endTime = time.time() - startTime print("time : ", endTime) pred_str = "" for i in range(len(recv_)): pred_str += str(pred[i]) print("result : ", pred_str) socket.send_string(pred_str) ```
wtnan2003 commented 4 years ago

@imsoo It works ！ thanks！

But It seems only got one result, then broken: image

./ventilator and ./worker seems to work fine

./sink broken image

All output Error: nd@nd-All-Series:~/fight_detection/server$ ./sink Sink | Recv From Worker | SEQ : 1 LEN : 74458 RNN fee: 3625.3ms | T : 1 Sink | Pub To Client | SEQ : 1 LEN : 77093 *** Error in `./sink': double free or corruption (!prev): 0x00007f9910000930 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f993e4f67e5] /lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f993e4ff37a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f993e50353c] ./sink[0x4284d5] ./sink[0x40796e] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f99428a56ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f993e58641d] ======= Memory map: ======== 00400000-00430000 r-xp 00000000 08:01 29180275 /home/nd/fight_detection/server/sink 0062f000-00630000 r--p 0002f000 08:01 29180275 /home/nd/fight_detection/server/sink 00630000-00631000 rw-p 00030000 08:01 29180275 /home/nd/fight_detection/server/sink 00631000-00632000 rw-p 00000000 00:00 0 00ba1000-01796000 rw-p 00000000 00:00 0 [heap] 10000000-10001000 rw-s 00000000 00:06 488 /dev/nvidia0 10001000-10002000 rw-s 00000000 00:06 488 /dev/nvidia0 10002000-10003000 rw-s 00000000 00:06 488 /dev/nvidia0 10003000-10004000 rw-s 00000000 00:06 488 /dev/nvidia0 10004000-10005000 rw-s 00000000 00:06 488 /dev/nvidia0 10005000-10006000 rw-s 00000000 00:06 488 /dev/nvidia0 10006000-10007000 rw-s 00000000 00:06 488 /dev/nvidia0 10007000-10008000 rw-s 00000000 00:06 488 /dev/nvidia0 10008000-10009000 rw-s 00000000 00:06 488 /dev/nvidia0 10009000-1000a000 rw-s 00000000 00:06 488 /dev/nvidia0 1000a000-1000b000 rw-s 00000000 00:06 488 /dev/nvidia0 1000b000-1000c000 rw-s 00000000 00:06 488 /dev/nvidia0 1000c000-1000d000 rw-s 00000000 00:06 488 /dev/nvidia0 1000d000-1000e000 rw-s 00000000 00:06 488 /dev/nvidia0 1000e000-1000f000 rw-s 00000000 00:06 488 /dev/nvidia0 1000f000-10010000 rw-s 00000000 00:06 488 /dev/nvidia0 10010000-20000000 ---p 00000000 00:00 0 200000000-200100000 rw-s 00000000 00:06 487 /dev/nvidiactl 200100000-200104000 rw-s 00000000 00:06 487 /dev/nvidiactl 200104000-200120000 ---p 00000000 00:00 0 200120000-200520000 rw-s 00000000 00:06 487 /dev/nvidiactl 200520000-200524000 rw-s 00000000 00:06 487 /dev/nvidiactl 200524000-200540000 ---p 00000000 00:00 0 200540000-200940000 rw-s 00000000 00:06 487 /dev/nvidiactl 200940000-200944000 rw-s 00000000 00:06 487 /dev/nvidiactl 200944000-200960000 ---p 00000000 00:00 0 200960000-200d60000 rw-s 00000000 00:06 487 /dev/nvidiactl 200d60000-200d64000 rw-s 00000000 00:06 487 /dev/nvidiactl 200d64000-200d80000 ---p 00000000 00:00 0 200d80000-201180000 rw-s 00000000 00:06 487 /dev/nvidiactl 201180000-201184000 rw-s 00000000 00:06 487 /dev/nvidiactl 201184000-2011a0000 ---p 00000000 00:00 0 2011a0000-2015a0000 rw-s 00000000 00:06 487 /dev/nvidiactl 2015a0000-2015a4000 rw-s 00000000 00:06 487 /dev/nvidiactl 2015a4000-2015c0000 ---p 00000000 00:00 0 2015c0000-2019c0000 rw-s 00000000 00:06 487 /dev/nvidiactl 2019c0000-2019c4000 rw-s 00000000 00:06 487 /dev/nvidiactl 2019c4000-2019e0000 ---p 00000000 00:00 0 2019e0000-201de0000 rw-s 00000000 00:06 487 /dev/nvidiactl 201de0000-201de4000 rw-s 00000000 00:06 487 /dev/nvidiactl 201de4000-201e00000 ---p 00000000 00:00 0 201e00000-202200000 rw-s 00000000 00:06 487 /dev/nvidiactl 202200000-202204000 rw-s 00000000 00:06 487 /dev/nvidiactl 202204000-202220000 ---p 00000000 00:00 0 202220000-202620000 rw-s 00000000 00:06 487 /dev/nvidiactl 202620000-202624000 rw-s 00000000 00:06 487 /dev/nvidiactl 202624000-202640000 ---p 00000000 00:00 0 202640000-202a40000 rw-s 00000000 00:06 487 /dev/nvidiactl 202a40000-202a44000 rw-s 00000000 00:06 487 /dev/nvidiactl 202a44000-202a60000 ---p 00000000 00:00 0 202a60000-202e60000 rw-s 00000000 00:06 487 /dev/nvidiactl 202e60000-202e64000 rw-s 00000000 00:06 487 /dev/nvidiactl 202e64000-202e80000 ---p 00000000 00:00 0 202e80000-203280000 rw-s 00000000 00:06 487 /dev/nvidiactl 203280000-203284000 rw-s 00000000 00:06 487 /dev/nvidiactl 203284000-2032a0000 ---p 00000000 00:00 0 2032a0000-2036a0000 rw-s 00000000 00:06 487 /dev/nvidiactl 2036a0000-2036a4000 rw-s 00000000 00:06 487 /dev/nvidiactl 2036a4000-2036c0000 ---p 00000000 00:00 0 2036c0000-203ac0000 rw-s 00000000 00:06 487 /dev/nvidiactl 203ac0000-203ac4000 rw-s 00000000 00:06 487 /dev/nvidiactl 203ac4000-203ae0000 ---p 00000000 00:00 0 203ae0000-203ee0000 rw-s 00000000 00:06 487 /dev/nvidiactl 203ee0000-203ee4000 rw-s 00000000 00:06 487 /dev/nvidiactl 203ee4000-203f00000 ---p 00000000 00:00 0 203f00000-204300000 rw-s 00000000 00:06 487 /dev/nvidiactl 204300000-204400000 rw-s 00000000 00:05 30333 /dev/zero (deleted) 204400000-204500000 rw-s 00000000 00:06 487 /dev/nvidiactl 204500000-204600000 rw-s 00000000 00:05 30334 /dev/zero (deleted) 204600000-204700000 rw-s 00000000 00:06 487 /dev/nvidiactl 204700000-204800000 rw-s 00000000 00:06 487 /dev/nvidiactl 204800000-204900000 rw-s 00000000 00:06 487 /dev/nvidiactl 204900000-2049e0000 rw-s 00000000 00:06 487 /dev/nvidiactl 2049e0000-f00000000 ---p 00000000 00:00 0 7f9900000000-7f990006d000 rw-p 00000000 00:00 0 7f990006d000-7f9904000000 ---p 00000000 00:00 0 7f99057fb000-7f99057fc000 ---p 00000000 00:00 0 7f99057fc000-7f9905ffc000 rwxp 00000000 00:00 0 7f9905ffc000-7f9905ffd000 ---p 00000000 00:00 0 7f9905ffd000-7f99067fd000 rwxp 00000000 00:00 0 7f99067fd000-7f99067fe000 ---p 00000000 00:00 0 7f99067fe000-7f9906ffe000 rwxp 00000000 00:00 0 7f9906ffe000-7f9906fff000 ---p 00000000 00:00 0 7f9906fff000-7f99077ff000 rwxp 00000000 00:00 0 7f99077ff000-7f9907800000 ---p 00000000 00:00 0 7f9907800000-7f9908000000 rwxp 00000000 00:00 0 7f9908000000-7f9908021000 rw-p 00000000 00:00 0 7f9908021000-7f990c000000 ---p 00000000 00:00 0 7f990c000000-7f990c021000 rw-p 00000000 00:00 0 7f990c021000-7f9910000000 ---p 00000000 00:00 0 7f9910000000-7f9910081000 rw-p 00000000 00:00 0 7f9910081000-7f9914000000 ---p 00000000 00:00 0 7f9914111000-7f9914112000 ---p 00000000 00:00 0 7f9914112000-7f9914912000 rwxp 00000000 00:00 0 7f9914912000-7f9915432000 r-xp 00000000 08:01 41158735 /usr/lib/x86_64-linux-gnu/libcuda.so.384.130 7f9915432000-7f9915631000 ---p 00b20000 08:01 41158735 /usr/lib/x86_64-linux-gnu/libcuda.so.384.130 7f9915631000-7f9915782000 rw-p 00b1f000 08:01 41158735 /usr/lib/x86_64-linux-gnu/libcuda.so.384.130 7f9915782000-7f9934000000 rw-p 00000000 00:00 0 7f9934000000-7f9934039000 rw-p 00000000 00:00 0 7f9934039000-7f9938000000 ---p 00000000 00:00 0 7f993840a000-7f9938440000 rw-p 00000000 00:00 0 7f9938440000-7f9938441000 ---p 00000000 00:00 0 7f9938441000-7f9938c41000 rwxp 00000000 00:00 0 7f9938c41000-7f9938c84000 r-xp 00000000 08:01 42211097 /usr/lib/nvidia-384/libnvidia-fatbinaryloader.so.384.130 7f9938c84000-7f9938e83000 ---p 00043000 08:01 42211097 /usr/lib/nvidia-384/libnvidia-fatbinaryloader.so.384.130 7f9938e83000-7f9938e8e000 rw-p 00042000 08:01 42211097 /usr/lib/nvidia-384/libnvidia-fatbinaryloader.so.384.130 7f9938e8e000-7f9938e93000 rw-p 00000000 00:00 0 7f9938e93000-7f993973a000 r-xp 00000000 08:01 41158770 /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.384.130 7f993973a000-7f9939939000 ---p 008a7000 08:01 41158770 /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.384.130 7f9939939000-7f9939a8f000 rw-p 008a6000 08:01 41158770 /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.384.130 7f9939a8f000-7f9939a9d000 rw-p 00000000 00:00 0 7f9939a9d000-7f9939aa2000 r-xp 00000000 08:01 43915944 /usr/local/cuda-9.0/lib64/libOpenCL.so.1.0.0 7f9939aa2000-7f9939ca1000 ---p 00005000 08:01 43915944 /usr/local/cuda-9.0/lib64/libOpenCL.so.1.0.0 7f9939ca1000-7f9939ca3000 rw-p 00004000 08:01 43915944 /usr/local/cuda-9.0/lib64/libOpenCL.so.1.0.0 7f9939ca3000-7f9939ea3000 rw-p 00000000 00:00 0 7f9939ea3000-7f9939ebc000 r-xp 00000000 08:01 41168546 /usr/lib/x86_64-linux-gnu/libtbbmalloc.so.2 7f9939ebc000-7f993a0bb000 ---p 00019000 08:01 41168546 /usr/lib/x86_64-linux-gnu/libtbbmalloc.so.2 7f993a0bb000-7f993a0bc000 r--p 00018000 08:01 41168546 /usr/lib/x86_64-linux-gnu/libtbbmalloc.so.2 7f993a0bc000-7f993a0be000 rw-p 00019000 08:01 41168546 /usr/lib/x86_64-linux-gnu/libtbbmalloc.so.2 7f993a0be000-7f993a0e1000 rw-p 00000000 00:00 0 7f993a0e1000-7f993a0e2000 ---p 00000000 00:00 0 7f993a0e2000-7f993a8e2000 rwxp 00000000 00:00 0 7f993a8e2000-7f993a8e3000 ---p 00000000 00:00 0 7f993a8e3000-7f993b0e3000 rwxp 00000000 00:00 0 7f993b0e3000-7f993b0e4000 ---p 00000000 00:00 0 7f993b0e4000-7f993b8e4000 rwxp 00000000 00:00 0 7f993b8e4000-7f993b8e5000 ---p 00000000 00:00 0 7f993b8e5000-7f993c0e5000 rwxp 00000000 00:00 0 7f993c0e5000-7f993c0ea000 r-xp 00000000 08:01 41164374 /usr/lib/x86_64-linux-gnu/libIlmThread-2_2.so.12.0.0 7f993c0ea000-7f993c2ea000 ---p 00005000 08:01 41164374 /usr/lib/x86_64-linux-gnu/libIlmThread-2_2.so.12.0.0 7f993c2ea000-7f993c2eb000 r--p 00005000 08:01 41164374 /usr/lib/x86_64-linux-gnu/libIlmThread-2_2.so.12.0.0 7f993c2eb000-7f993c2ec000 rw-p 00006000 08:01 41164374 /usr/lib/x86_64-linux-gnu/libIlmThread-2_2.so.12.0.0 7f993c2ec000-7f993c307000 r-xp 00000000 08:01 41164366 /usr/lib/x86_64-linux-gnu/libIex-2_2.so.12.0.0 7f993c307000-7f993c506000 ---p 0001b000 08:01 41164366 /usr/lib/x86_64-linux-gnu/libIex-2_2.so.12.0.0 7f993c506000-7f993c509000 r--p 0001a000 08:01 41164366 /usr/lib/x86_64-linux-gnu/libIex-2_2.so.12.0.0 7f993c509000-7f993c50a000 rw-p 0001d000 08:01 41164366 /usr/lib/x86_64-linux-gnu/libIex-2_2.so.12.0.0 7f993c50a000-7f993c515000 r-xp 00000000 08:01 41165248 /usr/lib/x86_64-linux-gnu/libjbig.so.0 7f993c515000-7f993c714000 ---p 0000b000 08:01 41165248 /usr/lib/x86_64-linux-gnu/libjbig.so.0 7f993c714000-7f993c715000 r--p 0000a000 08:01 41165248 /usr/lib/x86_64-linux-gnu/libjbig.so.0 7f993c715000-7f993c718000 rw-p 0000b000 08:01 41165248 /usr/lib/x86_64-linux-gnu/libjbig.so.0 7f993c718000-7f993c739000 r-xp 00000000 08:01 48239093 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f993c739000-7f993c938000 ---p 00021000 08:01 48239093 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f993c938000-7f993c939000 r--p 00020000 08:01 48239093 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f993c939000-7f993c93a000 rw-p 00021000 08:01 48239093 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7f993c93a000-7f993c99c000 r-xp 00000000 08:01 42751034 /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61 7f993c99c000-7f993cb9c000 ---p 00062000 08:01 42751034 /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61 7f993cb9c000-7f993cb9f000 rw-p 00062000 08:01 42751034 /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61 7f993cb9f000-7f993cba0000 rw-p 00000000 00:00 0 7f993cba0000-7f993cbb9000 r-xp 00000000 08:01 48239021 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f993cbb9000-7f993cdb8000 ---p 00019000 08:01 48239021 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f993cdb8000-7f993cdb9000 r--p 00018000 08:01 48239021 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f993cdb9000-7f993cdba000 rw-p 00019000 08:01 48239021 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f993cdba000-7f993cdfc000 r-xp 00000000 08:01 41164360 /usr/lib/x86_64-linux-gnu/libHalf.so.12.0.0 7f993cdfc000-7f993cffb000 ---p 00042000 08:01 41164360 /usr/lib/x86_64-linux-gnu/libHalf.so.12.0.0 7f993cffb000-7f993cffc000 r--p 00041000 08:01 41164360 /usr/lib/x86_64-linux-gnu/libHalf.so.12.0.0 7f993cffc000-7f993cffd000 rw-p 00042000 08:01 41164360 /usr/lib/x86_64-linux-gnu/libHalf.so.12.0.0 7f993cffd000-7f993d1c8000 r-xp 00000000 08:01 41156792 /usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.22.0.0 7f993d1c8000-7f993d3c8000 ---p 001cb000 08:01 41156792 /usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.22.0.0 7f993d3c8000-7f993d3cb000 r--p 001cb000 08:01 41156792 /usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.22.0.0 7f993d3cb000-7f993d4cc000 rw-p 001ce000 08:01 41156792 /usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.22.0.0 7f993d4cc000-7f993d4cd000 rw-p 00000000 00:00 0 7f993d4cd000-7f993d517000 r-xp 00000000 08:01 41158223 /usr/lib/x86_64-linux-gnu/libjasper.so.1.0.0 7f993d517000-7f993d716000 ---p 0004a000 08:01 41158223 /usr/lib/x86_64-linux-gnu/libjasper.so.1.0.0 7f993d716000-7f993d717000 r--p 00049000 08:01 41158223 /usr/lib/x86_64-linux-gnu/libjasper.so.1.0.0 7f993d717000-7f993d71b000 rw-p 0004a000 08:01 41158223 /usr/lib/x86_64-linux-gnu/libjasper.so.1.0.0 7f993d71b000-7f993d722000 rw-p 00000000 00:00 0 7f993d722000-7f993d793000 r-xp 00000000 08:01 41156667 /usr/lib/x86_64-linux-gnu/libtiff.so.5.2.4 7f993d793000-7f993d993000 ---p 00071000 08:01 41156667 /usr/lib/x86_64-linux-gnu/libtiff.so.5.2.4 7f993d993000-7f993d994000 r--p 00071000 08:01 41156667 /usr/lib/x86_64-linux-gnu/libtiff.so.5.2.4 7f993d994000-7f993d997000 rw-p 00072000 08:01 41156667 /usr/lib/x86_64-linux-gnu/libtiff.so.5.2.4 7f993d997000-7f993d9bb000 r-xp 00000000 08:01 48234501 /lib/x86_64-linux-gnu/libpng12.so.0.54.0 7f993d9bb000-7f993dbba000 ---p 00024000 08:01 48234501 /lib/x86_64-linux-gnu/libpng12.so.0.54.0 7f993dbba000-7f993dbbb000 r--p 00023000 08:01 48234501 /lib/x86_64-linux-gnu/libpng12.so.0.54.0 7f993dbbb000-7f993dbbc000 rw-p 00024000 08:01 48234501 /lib/x86_64-linux-gnu/libpng12.so.0.54.0 7f993dbbc000-7f993dc13000 r-xp 00000000 08:01 41156797 /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2 7f993dc13000-7f993de13000 ---p 00057000 08:01 41156797 /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2 7f993de13000-7f993de14000 r--p 00057000 08:01 41156797 /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2 7f993de14000-7f993de15000 rw-p 00058000 08:01 41156797 /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2 7f993de15000-7f993de18000 r-xp 00000000 08:01 48235548 /lib/x86_64-linux-gnu/libdl-2.23.so 7f993de18000-7f993e017000 ---p 00003000 08:01 48235548 /lib/x86_64-linux-gnu/libdl-2.23.so 7f993e017000-7f993e018000 r--p 00002000 08:01 48235548 /lib/x86_64-linux-gnu/libdl-2.23.so 7f993e018000-7f993e019000 rw-p 00003000 08:01 48235548 /lib/x86_64-linux-gnu/libdl-2.23.so 7f993e019000-7f993e020000 r-xp 00000000 08:01 48235566 /lib/x86_64-linux-gnu/librt-2.23.so 7f993e020000-7f993e21f000 ---p 00007000 08:01 48235566 /lib/x86_64-linux-gnu/librt-2.23.so 7f993e21f000-7f993e220000 r--p 00006000 08:01 48235566 /lib/x86_64-linux-gnu/librt-2.23.so 7f993e220000-7f993e221000 rw-p 00007000 08:01 48235566 /lib/x86_64-linux-gnu/librt-2.23.so 7f993e221000-7f993e27e000 r-xp 00000000 08:01 41170543 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1 7f993e27e000-7f993e47d000 ---p 0005d000 08:01 41170543 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1 7f993e47d000-7f993e47e000 r--p 0005c000 08:01 41170543 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1 7f993e47e000-7f993e47f000 rw-p 0005d000 08:01 41170543 /usr/lib/x86_64-linux-gnu/libsodium.so.18.0.1 7f993e47f000-7f993e63f000 r-xp 00000000 08:01 48235546 /lib/x86_64-linux-gnu/libc-2.23.so 7f993e63f000-7f993e83f000 ---p 001c0000 08:01 48235546 /lib/x86_64-linux-gnu/libc-2.23.so 7f993e83f000-7f993e843000 r--p 001c0000 08:01 48235546 /lib/x86_64-linux-gnu/libc-2.23.so 7f993e843000-7f993e845000 rw-p 001c4000 08:01 48235546 /lib/x86_64-linux-gnu/libc-2.23.so 7f993e845000-7f993e849000 rw-p 00000000 00:00 0 7f993e849000-7f993e85f000 r-xp 00000000 08:01 48239064 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f993e85f000-7f993ea5e000 ---p 00016000 08:01 48239064 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f993ea5e000-7f993ea5f000 rw-p 00015000 08:01 48239064 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f993ea5f000-7f993eb67000 r-xp 00000000 08:01 48235541 /lib/x86_64-linux-gnu/libm-2.23.so 7f993eb67000-7f993ed66000 ---p 00108000 08:01 48235541 /lib/x86_64-linux-gnu/libm-2.23.so 7f993ed66000-7f993ed67000 r--p 00107000 08:01 48235541 /lib/x86_64-linux-gnu/libm-2.23.so 7f993ed67000-7f993ed68000 rw-p 00108000 08:01 48235541 /lib/x86_64-linux-gnu/libm-2.23.so 7f993ed68000-7f993fbba000 r-xp 00000000 08:01 41697175 /usr/local/lib/libopencv_core.so.3.1.0 7f993fbba000-7f993fdba000 ---p 00e52000 08:01 41697175 /usr/local/lib/libopencv_core.so.3.1.0 7f993fdba000-7f993fdc3000 r--p 00e52000 08:01 41697175 /usr/local/lib/libopencv_core.so.3.1.0 7f993fdc3000-7f993fdf0000 rw-p 00e5b000 08:01 41697175 /usr/local/lib/libopencv_core.so.3.1.0 7f993fdf0000-7f993fdff000 rw-p 00000000 00:00 0 7f993fdff000-7f9941582000 r-xp 00000000 08:01 41697181 /usr/local/lib/libopencv_imgproc.so.3.1.0 7f9941582000-7f9941782000 ---p 01783000 08:01 41697181 /usr/local/lib/libopencv_imgproc.so.3.1.0 7f9941782000-7f994178a000 r--p 01783000 08:01 41697181 /usr/local/lib/libopencv_imgproc.so.3.1.0 7f994178a000-7f99417ac000 rw-p 0178b000 08:01 41697181 /usr/local/lib/libopencv_imgproc.so.3.1.0 7f99417ac000-7f9941842000 rw-p 00000000 00:00 0 7f9941842000-7f99419c4000 r-xp 00000000 08:01 41697185 /usr/local/lib/libopencv_video.so.3.1.0 7f99419c4000-7f9941bc4000 ---p 00182000 08:01 41697185 /usr/local/lib/libopencv_video.so.3.1.0 7f9941bc4000-7f9941bc5000 r--p 00182000 08:01 41697185 /usr/local/lib/libopencv_video.so.3.1.0 7f9941bc5000-7f9941bc8000 rw-p 00183000 08:01 41697185 /usr/local/lib/libopencv_video.so.3.1.0 7f9941bc8000-7f9941d7e000 r-xp 00000000 08:01 41697197 /usr/local/lib/libopencv_imgcodecs.so.3.1.0 7f9941d7e000-7f9941f7e000 ---p 001b6000 08:01 41697197 /usr/local/lib/libopencv_imgcodecs.so.3.1.0 7f9941f7e000-7f9941f80000 r--p 001b6000 08:01 41697197 /usr/local/lib/libopencv_imgcodecs.so.3.1.0 7f9941f80000-7f9941f83000 rw-p 001b8000 08:01 41697197 /usr/local/lib/libopencv_imgcodecs.so.3.1.0 7f9941f83000-7f9941f86000 rw-p 00000000 00:00 0 7f9941f86000-7f9941f99000 r-xp 00000000 08:01 41696707 /usr/local/lib/libjson-c.so.5.0.0 7f9941f99000-7f9942198000 ---p 00013000 08:01 41696707 /usr/local/lib/libjson-c.so.5.0.0 7f9942198000-7f9942199000 r--p 00012000 08:01 41696707 /usr/local/lib/libjson-c.so.5.0.0 7f9942199000-7f994219a000 rw-p 00013000 08:01 41696707 /usr/local/lib/libjson-c.so.5.0.0 7f994219a000-7f99421f8000 r-xp 00000000 08:01 41170331 /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.58.0 7f99421f8000-7f99423f7000 ---p 0005e000 08:01 41170331 /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.58.0 7f99423f7000-7f99423fa000 r--p 0005d000 08:01 41170331 /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.58.0 7f99423fa000-7f99423fb000 rw-p 00060000 08:01 41170331 /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.58.0 7f99423fb000-7f9942432000 r-xp 00000000 08:01 41168548 /usr/lib/x86_64-linux-gnu/libtbb.so.2 7f9942432000-7f9942632000 ---p 00037000 08:01 41168548 /usr/lib/x86_64-linux-gnu/libtbb.so.2 7f9942632000-7f9942633000 r--p 00037000 08:01 41168548 /usr/lib/x86_64-linux-gnu/libtbb.so.2 7f9942633000-7f9942635000 rw-p 00038000 08:01 41168548 /usr/lib/x86_64-linux-gnu/libtbb.so.2 7f9942635000-7f9942638000 rw-p 00000000 00:00 0 7f9942638000-7f994269a000 r-xp 00000000 08:01 41170649 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0 7f994269a000-7f9942899000 ---p 00062000 08:01 41170649 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0 7f9942899000-7f994289d000 r--p 00061000 08:01 41170649 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0 7f994289d000-7f994289e000 rw-p 00065000 08:01 41170649 /usr/lib/x86_64-linux-gnu/libzmq.so.5.0.0 7f994289e000-7f99428b6000 r-xp 00000000 08:01 48235545 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f99428b6000-7f9942ab5000 ---p 00018000 08:01 48235545 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f9942ab5000-7f9942ab6000 r--p 00017000 08:01 48235545 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f9942ab6000-7f9942ab7000 rw-p 00018000 08:01 48235545 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f9942ab7000-7f9942abb000 rw-p 00000000 00:00 0 7f9942abb000-7f9942c2d000 r-xp 00000000 08:01 41157000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f9942c2d000-7f9942e2d000 ---p 00172000 08:01 41157000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f9942e2d000-7f9942e37000 r--p 00172000 08:01 41157000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f9942e37000-7f9942e39000 rw-p 0017c000 08:01 41157000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f9942e39000-7f9942e3d000 rw-p 00000000 00:00 0 7f9942e3d000-7f9942e63000 r-xp 00000000 08:01 48235544 /lib/x86_64-linux-gnu/ld-2.23.so 7f9942e87000-7f9942e88000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9942e88000-7f9942e89000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9942e89000-7f9942e8a000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9942e8a000-7f9942e8b000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9942e8b000-7f9942eb3000 rw-p 00000000 00:00 0 7f9942f94000-7f9943036000 rw-p 00000000 00:00 0 7f9943036000-7f9943037000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9943037000-7f9943038000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9943038000-7f9943039000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9943039000-7f994303a000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f994303a000-7f994303b000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f994303b000-7f994303c000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f994303c000-7f994303d000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f994303d000-7f994303e000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f994303e000-7f994303f000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f994303f000-7f9943040000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9943040000-7f9943041000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9943041000-7f9943042000 rw-s 00000000 00:06 487 /dev/nvidiactl 7f9943042000-7f9943062000 rw-p 00000000 00:00 0 7f9943062000-7f9943063000 r--p 00025000 08:01 48235544 /lib/x86_64-linux-gnu/ld-2.23.so 7f9943063000-7f9943064000 rw-p 00026000 08:01 48235544 /lib/x86_64-linux-gnu/ld-2.23.so 7f9943064000-7f9943065000 rw-p 00000000 00:00 0 7ffe8de15000-7ffe8de35000 rwxp 00000000 00:00 0 [stack] 7ffe8de35000-7ffe8de36000 rw-p 00000000 00:00 0 7ffe8decb000-7ffe8dece000 r--p 00000000 00:00 0 [vvar] 7ffe8dece000-7ffe8ded0000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted (core dumped)
imsoo commented 4 years ago

@wtnan2003

Could you change MSG_BUF_LEN( 76800 -> 102400) and try again? (need to rebuild all)

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/server/src/frame.hpp#L16

wtnan2003 commented 4 years ago

@imsoo thanks for your quick reply!!

Sink doesn't break now after rebuild all

But: *** Error in `./darknet_client': free(): corrupted unsorted chunks: 0xR : 0 | C : 542 | F : 0 | T : 543 : 1 R : 0 | C : 542 | F : 0 | T : R : 0 | C : 542 | F : 0 | T : 543 : 1 Aborted (core dumped)

Snipaste_2020-06-07_16-19-38

imsoo commented 4 years ago

@wtnan2003

Sorry, a client needs change too.

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/client/darknet_client/src/frame.hpp#L15

wtnan2003 commented 4 years ago

@imsoo Thanks for your help! Now Everything works fine, not Error any more but output_video file_size is increasing very slowly image I will check it when finished

imsoo commented 4 years ago

@wtnan2003

Thanks for your help! Now Everything works fine, not Error any more

:+1:

but output_video file_size increasing very slowly

Did you set GPU option when you build a darknet library?

# AlexeyAB/darknet/Makefile
GPU=1
CUDNN=1
CUDNN_HALF=0
OPENCV=1
AVX=0
OPENMP=1
LIBSO=1
ZED_CAMERA=0 # ZED SDK 3.0 and above
ZED_CAMERA_v2_8=0 # ZED SDK 2.X
wtnan2003 commented 4 years ago

@imsoo oh... I forgot to do that 😂

wtnan2003 commented 4 years ago

@imsoo I check the video output (ignore the wrong pixel, which caused kick by gif recorder):

there are not "kick" detected, anything wrong?

imsoo commented 4 years ago

@wtnan2003

Sorry, action recognition performance is not good especially kick action.

I think it needs more train. (with dataset and add more feature vector)

But try below thing to improve result.

  1. Change width, height and try again. (200 -> 400)

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/server/cfg/openpose.cfg#L2-L3

  1. Try action2.h5. (action.h5 -> action2.h5)

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/server/action.py#L26

wtnan2003 commented 4 years ago

thx a lot🤝

I will try it later

wtnan2003 commented 4 years ago

@imsoo

Everything works fine when I run ./darknet_client ... video1.mp4 ...

But I get empty output_video file_size when I want to continue to detect another video (by simply run ./darknet_client ... video2.mp4 ... without restart other servers)

It seems to disconnect between client and action when I finish a video?

thx!

imsoo commented 4 years ago

@wtnan2003

Oh sorry, I forgot to tell you.

After one video is over, we need to re-run sink.

The sink retains the track_frame to send frames of the video to the client in order.

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/server/src/sink.cpp#L140

So after one video is over, track_frame be a total number of frames.

But sink doesn't know if it's the end. This causes a problem.

If the client starts sending frames of another video, sink think that this frame is not the right order.

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/server/src/sink.cpp#L159-L169

To solve this problem, I think we need to add something to mark the end of the video.

This is my TODO but I haven't done it yet.

wtnan2003 commented 4 years ago

@imsoo thx anyway

By the way，what is the meaning of the number of bounding boxes?

image

imsoo commented 4 years ago

It's an ID given as a result of object(person) tracking.

wtnan2003 commented 4 years ago

hi @imsoo ，I notice that fight detection is base on:

This stage check that person who kick or punch is hitting someone. if some person has hit other, those people set enemy each other. System count it as fight and then track them until they exist in frame.

I have some questions:

  1. how to define which person is being punched or kicked (If there are many people in one frame)?
  2. what is meaning of until they exist in frame.，It seems to track many frames。Is it track them until tracker loses one of the fighters?

I would be appreciated if you help me out of these

imsoo commented 4 years ago

@wtnan2003

1.how to define which person is being punched or kicked (If there are many people in one frame)?

In Sink process, We run this pipeline.

Tracking (Tracker) -> Action Recogintion (Action) -> Fight Detection (In Sink)

Tracker receive bounding boxes (is produced which is a result of pose estimation) and produces object unique IDs. Plus, Tracker manage time series data about each person.

So we can get unique data (joint time series data) about each person. This data is used to recognize the action of each person.

For instance, (not the actual outcome.)

Stage Frame 12 Frame 32
Input
Pose
Track
Action

2.what is meaning of until they exist in frame.，It seems to track many frames。Is it track them until tracker loses one of the fighters?

Yes, It's when a tracker misses them or they disappear.

wtnan2003 commented 4 years ago

@imsoo
Thank you for your explanation.

  1. Each person got a unique ID and data. For instance, there are three people, marked: ID1、ID2、ID3.
  2. The action of each person is ——ID1： walking、ID2：standing、ID3：punching.
  3. My problem is how to know who is ID3 enemy ？

how Crash detention work？ （maybe bounding box IOU？） image

imsoo commented 4 years ago

@wtnan2003

Compare the position of the wrist and elbow with the other person's bounding box. If the wrist and elbow are located in the bounding box, it's a crash. (set as an enemy to each other) For kick action, use the position of the ankles and knees.

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/server/src/people.cpp#L147-L175

In Sink process, The comparison is a simply nested for loop.

For instance,

ID1： walking、ID2：standing、ID3：punching.

ID1 : Walking (Do not check) ID2 : Standing (Do not check) ID3 : Punching ----> Check (comparison with ID1) ----> Check (comparison with ID2)

https://github.com/imsoo/fight_detection/blob/f6a435ce03bda03d1f722058c1fce44b231da254/server/src/sink.cpp#L268-L284