_run_in_batches Issue With Multi Processing

[fspider]

• I am using darknet for detection, deepsort for tracking, mtcnn for age-gender detection It works fine with one function in main
• But when run this function with multi processing. there is Issue. When i run only one process with multiprocessing, it is the same.
• The issue is in deepsort which is on waiting status with _run_in_batches function in generate_detection.py/class ImageEncoder/ def call/_run_in_batches()

I found this issue with log

Code is like this

def worker(FLAGS, pno, lock):
    model_filename = 'model_data/market1501.pb'
    encoder = gdet.create_box_encoder(model_filename, batch_size=1)
    while:
         darknet_detection part
         boxs, confidences, classids = generate_boxes_confidences_classids(results, FLAGS.confidence)
         if (boxs is not None):
             features = encoder(frame, boxs)
             ....

if __name__ == '__main__':
    jobs = []
    p = Process(target=worker, args=(FLAGS, 0, lock))
    jobs.append(p)
    p.start()

generate_detection.py

def __call__(self, data_x, batch_size=32):
    print("->190")
    out = np.zeros((len(data_x), self.feature_dim), np.float32)
    print("->191")
    _run_in_batches(
        lambda x: self.session.run(self.output_var, feed_dict=x),
        {self.input_var: data_x}, out, batch_size)
    print("->192")
    return out

ENVIRONMENT

Ubuntu 18.04, Cuda 10.0, Tensorflow 1.14

Log RESULT

->13
->16
->17
->18
->19
->190
->191

So i found the _run_in_batches function not works with multiprocessing

How can i run deep sort module with multiprocessing?

[fspider]

And when trace the log with _run_in_baches function i get this log result

def _run_in_batches(f, data_dict, out, batch_size):
    data_len = len(out)
    num_batches = int(data_len / batch_size)
    print("123")
    s, e = 0, 0
    print("124")
    for i in range(num_batches):
        print("1240")
        s, e = i * batch_size, (i + 1) * batch_size
        print("1241")
        batch_data_dict = {k: v[s:e] for k, v in data_dict.items()}
        print("1242")
        out[s:e] = f(batch_data_dict)
        print("1243")
    print("125")
    if e < len(out):
        print("1250")
        batch_data_dict = {k: v[e:] for k, v in data_dict.items()}
        print("1251")
        out[e:] = f(batch_data_dict)
        print("1252")

Log RESULT

123
124
1240
1241
1242

So it seems not work with following function out[s:e] = f(batch_data_dict)

Please Help me!

[songzhifei]

i have the same problem . how do you solve it?

[gustavojoseleite]

Same problem here! Any solution?