Running for a long time and get no result from NCS

[zree]

Hi, I got a weird problem while I'm trying to implement two difference tasks on NCS2 like you did in this work.

I bind a detection net and a recognition net to one NCS2, using async mode. It did work, but I found that if I set num_request for each net to 4 and run for some time, such like 10 minutes, the device status return from Wait() function will always be -9, which means the result is not ready. if I set num_request to 2, it can run for a longer time, such like 3hours, and if set to 1, it can run normally. Have you ever tried to run your program for a long time? I don't know if the problem is about overload..

I noticed that you only call the IEPlugin() function once, and bind those nets to the same plugin no matter how many NCSs you use, the only difference caused by the NCS number is the num_requests, am I right? Does it mean OpenVINO will automatically allocate those work on those NCS2?

[PINTO0309]

uch like 3hours, and if set to 1, it can run normally. Have you ever tried to run your program for a long time? I don't know if the problem is about overload..

Yes. I know the problem that NCS2 becomes unstable with long operation. The NCS2 is not suitable for driving for a long time. Maximizing performance will shorten the continuous drive time. Incidentally, there is a specification that hangs up when the internal temperature of NCS reaches 70 ° C to 75 ° C. If long-term driving is required, you should use NCSDK instead of OpenVINO.

Does it mean OpenVINO will automatically allocate those work on those NCS2?

Yes, Exactly. However, OpenVINO can not control NCS assignment by itself.

It is similar to the answer to the previous question, but maximizing performance shortens the time that normal drives are available. You need to adjust "num_requests" according to your operational requirements. Alternatively, you can use fine-grained control of one NCS unit, using the low level API of NCSDK. However, unfortunately, NCSDK does not support NCS2. I am strongly hoping that NCSDK will correspond to NCS2 by nature.

[zree]

That's exactly what I'm confused about. Thank you so much for this detailed explanation! But I still find some points I couldn't understand completely.

If long-term driving is required, you should use NCSDK instead of OpenVINO. Alternatively, you can use fine-grained control of one NCS unit, using the low level API of NCSDK.

1. Why is NCSDK more suitable for long-term driving? I thought NCSDK is integrated into OpenVINO. Could you give more info about what's the advantage of NCSDK?
2. I tried NCSDK before, I got that it provides a sync mode for inference and can load 2 different nets simultaneously. The only method to deploy a multi-NCS program that I can imagine is to implement the whole processing in one thread and create threads as many as NCSs. I experimented on multithread before, just to find that more threads will bring more thread switch, and the speed will slow down. Have you encountered this problem before? Is there an async mode that NCSDK can provide?
3. If there's no async mode for NCSDK, does it mean sync mode in OpenVINO will perform the same as NCSDK?

[PINTO0309]

Why is NCSDK more suitable for long-term driving? I thought NCSDK is integrated into OpenVINO. Could you give more info about what's the advantage of NCSDK?

Strictly speaking, OpenVINO and NCSDK differ in the types of APIs provided. For example, NCSDK. Multiple NCS can be controlled individually by API. But, OpenVINO can not control individually.

    for device in devices:
        try:
            device = mvnc.Device(device)
            device.open()
            for (graph, graph_buffer) in zip(graphs, graph_buffers):
                graphHandles.append(graph.allocate_with_fifos(device, graph_buffer))
            devopen = True
            break
        except:
            continue

To improve the stability of NCS, I implemented a unique mechanism to distribute the load to multiple NCS using Python. https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiStickSSDwithRealSense.py Also, from the perspective of maintaining stability, I implemented an algorithm to know the internal temperature of NCS in detail, and to stop from the Python program side when the temperature exceeds a certain temperature.

The only method to deploy a multi-NCS program that I can imagine is to implement the whole processing in one thread and create threads as many as NCSs. I experimented on multithread before, just to find that more threads will bring more thread switch, and the speed will slow down. Have you encountered this problem before? Is there an async mode that NCSDK can provide?

I implemented with "MultiProcess" which performs better than "MultiThread". MultiThread does not improve performance due to the Python "GIL (Global Interpreter Lock)" issue. Operates up to about 4 times faster. NCS x4 boosted. One process is assigned to one NCS to operate at high speed. https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiStickSSDwithRealSense.py

If there's no async mode for NCSDK, does it mean sync mode in OpenVINO will perform the same as NCSDK?

My implementation has already been implemented as "asynchronous + synchronous".

Inference = asynchronous processing (MultiStick support by MultiProcess) Movie shooting = synchronous Inference + Movie shooting = asynchronous

Since I only implemented a simple implementation like a toy, I need to consider an algorithm for securing the order between frames. In the following program, the mechanism for securing the order between frames is realized using "heapqueue". https://github.com/PINTO0309/MobileNet-SSD-RealSense/blob/master/MultiStickSSDwithUSBCamera_OpenVINO_NCS2.py

[zree]

Oh I think I already got this. Thank you so much!