search

decemberpei / openvino-ncs2-python-samples

OpenVINO Inference Engine Python API sample code - NCS2
MIT License
34 stars 8 forks source link

Can NCS2 use multiple processes? Or can an NCS2 run multiple models simultaneously? #2

Open ZhaoMonica opened 5 years ago

ZhaoMonica commented 5 years ago

Hello, may I ask "_worker = threading. Thread (target = async_infer_worker, args= (exec_nets [f], request_number, image_queue, out_blob)" in your "async_api_multi-processes_multi-requests_multi-ncs.py" file?

_ worker. start ()

Infer_threads. append (_worker)

Can "preprocess_process. join ()" be changed to process run? I tried unsuccessfully myself. I only have one NCS2. I want to run two models, one Yolo and one SSD model at the same time. Does NCS2 support it?

decemberpei commented 5 years ago

am sorry your post is badly formatted i can't catch your meaning. For your last question, The answer is Yes. you can run two models simutaneously with only one NCS2, with lower performance result of course.

ZhaoMonica commented 5 years ago

from openvino.inference_engine import IENetwork, IEPlugin import cv2 import multiprocessing as mp

def main(net, plugin):

input_blob = next(iter(net.inputs))
out_blob = next(iter(net.outputs))

exec_net = plugin.load(network=net)
frame = cv2.imread("/home/sdu/my_test/1557038029.0015957.jpeg")
n, c, h, w = net.inputs[input_blob].shape

image = cv2.resize(frame, (w, h))
image = image.transpose((2, 0, 1))
image = image.reshape((n, c, h, w))

res = exec_net.infer({input_blob: image})
output_blob = res
print(output_blob[out_blob].shape)

##################################################### def run(): net1 = IENetwork( model='/home/sdu/my_test/model_data/yolov2/FP16_new/yolo_test.xml', weights='/home/sdu/my_test/model_data/yolov2/FP16_new/yolov_test.bin') net2 = IENetwork(model='/home/sdu/my_test/model_data/ssd/ssd/deploy.xml', weights='/home/sdu/my_test/model_data/ssd/ssd/deploy.bin')

plugin = IEPlugin(device="MYRIAD")
nets = [net1, net2]
processes = []
for net in nets:
    processes.append(mp.Process(target=main, args=(net,plugin)))

[setattr(process, "daemon", True) for process in processes]  # process.daemon = True
[process.start() for process in processes]
[process.join() for process in processes]
del net
del plugin

run()

ZhaoMonica commented 5 years ago

I use the above code and run two models at the same time to report the following errors:

Process Process-1:

Traceback (most recent call last):

File "/usr/lib/python 3.5/multiprocessing/process.py", line 249, in_bootstrap

Self.run ()

File "/usr/lib/python 3.5/multiprocessing/process.py", line 93, in run

Self. _target (* self. _args,** self. _kwargs)

File "mulit_nets_test_01.py", line 12, in main

Exec_net = plugin. load (network = net)

File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load

File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load

Runtime Error: Can not init USB device: NC_DEVICE_NOT_FOUND

E: [ncAPI] [719106] ncDeviceOpen: 870 failed to find device

Process Process-2:

Traceback (most recent call last):

File "/usr/lib/python 3.5/multiprocessing/process.py", line 249, in_bootstrap

Self.run ()

File "/usr/lib/python 3.5/multiprocessing/process.py", line 93, in run

Self. _target (* self. _args,** self. _kwargs)

File "mulit_nets_test_01.py", line 12, in main

Exec_net = plugin. load (network = net)

File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load

File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load

Runtime Error: Can not init USB device: NC_DEVICE_NOT_FOUND

In fact, I can be sure that my NCS2 is plugged in and there is no problem. If I use "processes. append" (threading. Thread (target = main, args = (net, plugin)) to get results, does this mean that NCS2 cannot run two models in parallel?

ZhaoMonica commented 5 years ago

processes.append(threading.Thread(target=main, args=(net,plugin )))

decemberpei commented 5 years ago

if you are sure the issue is not caused by your NCS2, suggest you try use threads instead of processes ( since different processed have different memory spaces, "passing" IEPlugin objects between processes may cause unexpected behavior like what you've ran into).

if you really want multiple processes, you can init IEPlugin instances in child processes. this should also work.

ZhaoMonica commented 5 years ago

Hello, I don't quite understand what you mean by "Initializing IEPlugin in a child process". Can you show it in my code above? Thank you very much.

ZhaoMonica commented 5 years ago

[<Process(Process-1, initial)>, <Process(Process-2, initial)>] E: [ncAPI] [ 175625] ncDeviceOpen:870 failed to find device

Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "mulit_nets_test_01.py", line 12, in main exec_net = plugin.load(network=net) File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load RuntimeError: Can not init USB device: NC_DEVICE_NOT_FOUND (1, 1, 100, 7)

I put the initialization of IEPlugin in the main () function of the code above, which shows the results. Only one model has been loaded and output. Do you know why?

decemberpei commented 5 years ago
  1. use threads , not processes ( in your code you fork 2 processes and do job in child processes. please fork threads instead. )
  2. init only one IEPlugin in main thread, and pass it to sub-threads.

ZhaoMonica notifications@github.com于2019年5月23日 周四下午5:23写道：

[<Process(Process-1, initial)>, <Process(Process-2, initial)>] E: [ncAPI] [ 175625] ncDeviceOpen:870 failed to find device Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "mulit_nets_test_01.py", line 12, in main exec_net = plugin.load(network=net) File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load RuntimeError: Can not init USB device: NC_DEVICE_NOT_FOUND (1, 1, 100, 7)

I put the initialization of IEPlugin in the main () function of the code above, which shows the results. Only one model has been loaded and output. Do you know why?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/decemberpei/openvino-ncs2-python-samples/issues/2?email_source=notifications&email_token=AAJ2K2VSP34GQDXQE3QIZB3PWZO77A5CNFSM4HO2PROKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWBTSLY#issuecomment-495139119, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ2K2VWQXFONWAOVLFRDX3PWZO77ANCNFSM4HO2PROA .

--

Wechat: decemberpei Cell Phone: (86)176-1216-2932

ZhaoMonica commented 5 years ago

Hello, can I understand that a NCS2 can only perform "plugin = IEPlugin" (device = "MYRIAD") once, and if it is a process, it will need several NCS2 to initialize IEPlugin once for each process.

decemberpei commented 5 years ago

Hello, can I understand that a NCS2 can only perform "plugin = IEPlugin" (device = "MYRIAD") once, and if it is a process, it will need several NCS2 to initialize IEPlugin once for each process.

I haven't tried but I don't agree,

  1. If in your OS (say, Linux/Windows), there are several separate apps which are trying to use the only one NCS2, probably they don't know the existance of each other, and will init their own IEPlugin instance. it would be very confusing if one App succeed to init IEPlugin while others fail to do so.
  2. from OS point of view, "processes" are just different Apps that run in parallel, when you pass objects such as IEPlugin instances to other processes, the object get "cloned", the cloned object is by no mean the original one and may cause errors when you try to invoke it. hence If I were you i won't pass IEPlugin instances between processes.
ZhaoMonica commented 5 years ago

I have some questions about your first point: if there are several separate applications trying to use the same NCS2, does NCS2 support separate access for several separate applications?

ZhaoMonica commented 5 years ago

I use process simulation to simulate several separate applications to access NCS2 at the same time. In fact, when I insert an NCS2, only one application is working properly. Others will "fail to find NCS2". When I insert two NCS2, two programs will run normally. Does this mean that NCS2 cannot be processed or accessed simultaneously?

decemberpei commented 5 years ago

I know some one reporting the same issue. I haven't tried this yet. my guess is that one NCS can be bond to only one IEPlugin.

ZhaoMonica notifications@github.com于2019年5月24日 周五下午5:38写道：

I use process simulation to simulate several separate applications to access NCS2 at the same time. In fact, when I insert an NCS2, only one application is working properly. Others will "fail to find NCS2". When I insert two NCS2, two programs will run normally. Does this mean that NCS2 cannot be processed or accessed simultaneously?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/decemberpei/openvino-ncs2-python-samples/issues/2?email_source=notifications&email_token=AAJ2K2S6F3ZWJBP5N4SMTFTPW6ZSZA5CNFSM4HO2PROKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWEWT4Q#issuecomment-495544818, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ2K2VSWODENI4CGETYUH3PW6ZSZANCNFSM4HO2PROA .

--

Wechat: decemberpei Cell Phone: (86)176-1216-2932

decemberpei commented 5 years ago

I can't find an answer for your question from official document. it may or may not be supported.

ZhaoMonica notifications@github.com于2019年5月24日 周五下午5:31写道：

I have some questions about your first point: if there are several separate applications trying to use the same NCS2, does NCS2 support separate access for several separate applications?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/decemberpei/openvino-ncs2-python-samples/issues/2?email_source=notifications&email_token=AAJ2K2UUMX66L5X4FMGQOZLPW6YWFA5CNFSM4HO2PROKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWEV4FQ#issuecomment-495541782, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJ2K2VVY6TCN7MXBBOQTMLPW6YWFANCNFSM4HO2PROA .

--

Wechat: decemberpei Cell Phone: (86)176-1216-2932

ZhaoMonica commented 5 years ago

Hi, how can I avoid having only one thread running if I have an NCS2 and start four threads, each of which is an infinite loop?I want all four threads to execute in rotation, not just one running and all the others blocking.

decemberpei commented 5 years ago

Hi, how can I avoid having only one thread running if I have an NCS2 and start four threads, each of which is an infinite loop?I want all four threads to execute in rotation, not just one running and all the others blocking.

sorry your question has nothing to do with NCS2 nor this repo, Please google it yourself.

decemberpei commented 5 years ago

just found some tips in official doc: http://docs.openvinotoolkit.org/latest/_docs_IE_DG_supported_plugins_MYRIAD.html#MYRIAD_DEVICE_ALLOC