search

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
https://docs.openvino.ai
Apache License 2.0
7.17k stars 2.24k forks source link

Custom IR model does not work in MYRIAD #270

Closed katakissof closed 2 years ago

katakissof commented 5 years ago

Hello, I have trained a model in VOC with the pytorch toolkit of the openvino training extensions and I converted into onnx format with the provided utilities of the integrated nncf package. Afterwards, I converted the onnx file it into the IR representation and ran the benchmark app of the openvino. It worked in the CPU but failed in the MYRIAD plugin with the following error:

[ ERROR ] AssertionFailed: data->numConsumers() > 0

I attach you the zip file with the IR model and the log file of the benchmark app. Can you explain me why it fails ?

Thank you in advance. info.zip

shubha-ramani commented 5 years ago

Dear @katakissof Thanks for your attached .zip. However, I'm going to need your original model as well as your inference script (please make sure the inference script is a simple .py program).

Thanks,

Shubha

katakissof commented 5 years ago

Dear @shubha-ramani , Thanks for your quick answer. You can find all the related models with the inference code that produces me the above-mentioned error in the dropbox zip file below.

https://www.dropbox.com/s/vodr2snbkd9fwht/material.zip?dl=0

shubha-ramani commented 5 years ago

Dear @katakissof Actually dropbox is inconvenient. Many of us do not have dropbox accounts. Is there anyway you can attach a *.zip again to this ticket ? If you cannot do that, let me know your email address and we can provide a way for you to transfer the files via Syncplicity.

Also kindly make sure that the code in your material.zip is as simple as possible which demonstrates the problem. For instance, please don't provide huge projects which we have to parse through. The reason for this is because we are dealing with tons of customers and we don't have the time to dig through everyone's complex projects.

Hope you understand.

Please kindly attach a *.zip if you can.

Sincerely,

Shubha

SofoklisKat commented 5 years ago

Dear @shubha-ramani , Thanks a lot for your answer. The dropbox folder is pretty big in size (163 mb) since I provided you a variety of models (onnx, pth and IR) and information in order to facilitate you with the debugging. But following your request I attach you a smaller .zip file with the necessary files (IR and inference code) only. If you need also the other files please let me know it to exchange emails and send you the extra files via Syncplicity. Thanks, Sofoklis Info.zip

mingzhaochina commented 4 years ago

Hello,I traced this issue from intel forum to here,it was said the new version will solve this problem,so I am curious about when?So far we can still only use CPU...

brmarkus commented 4 years ago

When using the attached model "info_small/FP16/model_FP16.xml" with the "benchmark_app" then I can confirm no errors, no warnings, no exceptions occur (anymore):

$> inference_engine_samples/intel64/Release/benchmark_app -m /defect/info_small/FP16/model_FP16.xml -d MYRIAD
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance,but it still may be non-optimal for some cases, for more information look at README.

[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
        API version ............ 2.1
        Build .................. 37588
        Description ....... API
[ INFO ] Device info:
        MYRIAD
        myriadPlugin version ......... 2.1
        Build ........... 37588

[Step 3/11] Setting device configuration
[Step 4/11] Reading the Intermediate Representation network
[ INFO ] Loading network files
[ INFO ] Read network took 12.14 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1, precision: MIXED
[Step 6/11] Configuring input of the model
[Step 7/11] Loading the model to the device
[ INFO ] Load network took 1935.19 ms
[Step 8/11] Setting optimal runtime parameters
[Step 9/11] Creating infer requests and filling input blobs with images
[ INFO ] Network input 'input.1' precision U8, dimensions (NCHW): 1 3 300 300
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Infer Request 0 filling
[ INFO ] Fill input 'input.1' with random values (image is expected)
[ INFO ] Infer Request 1 filling
[ INFO ] Fill input 'input.1' with random values (image is expected)
[ INFO ] Infer Request 2 filling
[ INFO ] Fill input 'input.1' with random values (image is expected)
[ INFO ] Infer Request 3 filling
[ INFO ] Fill input 'input.1' with random values (image is expected)
[Step 10/11] Measuring performance (Start inference asyncronously, 4 inference requests, limits: 60000 ms duration)

[Step 11/11] Dumping statistics report
Count:      3252 iterations
Duration:   60084.70 ms
Latency:    73.84 ms
Throughput: 54.12 FPS
Peak Virtual Memory (VmPeak) Size, kBytes: 1931492
Peak Resident Memory (VmHWM) Size, kBytes:  85436
mingzhaochina commented 4 years ago

This is mine:

$ python3 /home/zm/intel/openvino_2019.3.376/deployment_tools/tools/benchmark_tool/benchmark_app.py -m inference_graph.xml -d CPU [Step 1/11] Parsing and validating input arguments [ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README. [Step 2/11] Loading Inference Engine [ INFO ] InferenceEngine: API version............. 2.1.custom_releases/2019/R3_ac8584cb714a697a12f1f30b7a3b78a5b9ac5e05 [ INFO ] Device info CPU MKLDNNPlugin............ version 2.1 Build................... 32974

[Step 3/11] Reading the Intermediate Representation network [Step 4/11] Resizing network to match image sizes and given batch [ INFO ] Network batch size: 1, precision: FP16 [Step 5/11] Configuring input of the model [Step 6/11] Setting device configuration [Step 7/11] Loading the model to the device [Step 8/11] Setting optimal runtime parameters [Step 9/11] Creating infer requests and filling input blobs with images [ INFO ] Network input 'X' precision U8, dimensions (NCHW): 1 3 3000 1 [ WARNING ] No input files were given: all inputs will be filled with random values! [ INFO ] Infer Request 0 filling [ INFO ] Fill input 'X' with random values (image is expected) [ INFO ] Infer Request 1 filling [ INFO ] Fill input 'X' with random values (image is expected) [ INFO ] Infer Request 2 filling [ INFO ] Fill input 'X' with random values (image is expected) [ INFO ] Infer Request 3 filling [ INFO ] Fill input 'X' with random values (image is expected) [Step 10/11] Measuring performance (Start inference asyncronously, 4 inference requests using 4 streams for CPU, limits: 60000 ms duration) [Step 11/11] Dumping statistics report Count: 72392 iterations Duration: 60014.94 ms Latency: 3.18 ms Throughput: 1206.23 FPS

and

$ python3 /home/zm/intel/openvino_2019.3.376/deployment_tools/tools/benchmark_tool/benchmark_app.py -m inference_graph.xml -d MYRIAD [Step 1/11] Parsing and validating input arguments [ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README. [Step 2/11] Loading Inference Engine [ INFO ] InferenceEngine: API version............. 2.1.custom_releases/2019/R3_ac8584cb714a697a12f1f30b7a3b78a5b9ac5e05 [ INFO ] Device info MYRIAD myriadPlugin............ version 2.1 Build................... 32974

[Step 3/11] Reading the Intermediate Representation network [Step 4/11] Resizing network to match image sizes and given batch [ INFO ] Network batch size: 1, precision: FP16 [Step 5/11] Configuring input of the model [Step 6/11] Setting device configuration [Step 7/11] Loading the model to the device [Info ][VPU][MyriadPlugin] Device #0 MYRIAD-X (USB protocol) allocated [ ERROR ] AssertionFailed: _allocatedIntermData.count(topParent) > 0 Traceback (most recent call last): File "/home/zm/intel/openvino_2019.3.376/deployment_tools/tools/benchmark_tool/benchmark_app.py", line 82, in main exe_network = benchmark.load_network(ie_network, perf_counts, args.number_infer_requests) File "/opt/intel/openvino_2019.3.376/python/python3.6/openvino/tools/benchmark/benchmark.py", line 127, in load_network num_requests=number_infer_requests or 0) File "ie_api.pyx", line 85, in openvino.inference_engine.ie_api.IECore.load_network File "ie_api.pyx", line 92, in openvino.inference_engine.ie_api.IECore.load_network RuntimeError: AssertionFailed: _allocatedIntermData.count(topParent) > 0

model.tar.gz

brmarkus commented 4 years ago

@mingzhaochina using your attached model "inference_graph.xml" I also can't get the network loaded to the MyriadX (NCS2) (connected via USB3, on Ubuntu 16.04 LTS). It's working with the initially attached models, but not with yours...

mingzhaochina commented 4 years ago

This seems to be a problem at the IR process,the IR will generate four outputs,1715/Split.0, 1715/Split.2, 1725/Split.1, cross_entropy/logits,only "cross_entropy/logits" is mine.For CPU it can still work through,but for MYRIAD these three fake outputs will cause problems...

brmarkus notifications@github.com 于2020年1月22日周三 上午12:54写道：

@mingzhaochina https://github.com/mingzhaochina using your attached model "inference_graph.xml" I also can't get the network loaded to the MyriadX (NCS2) (connected via USB3, on Ubuntu 16.04 LTS). It's working with the initially attached models, but not with yours...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencv/dldt/issues/270?email_source=notifications&email_token=AGALZBVQBYRQV5PO5NC4CHTQ64SC5A5CNFSM4I5EC4Z2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJQOIGQ#issuecomment-576775194, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGALZBW2PMMIDXBSIRHI3WLQ64SC5ANCNFSM4I5EC4ZQ .

mingzhaochina commented 4 years ago

Hello, Would you please test this new model for me?Many thanks! @brmarkus $ /home/zm/dldt/inference-engine/bin/intel64/Release/benchmark_app -m FP16/inference_graph.xml -d MYRIAD [Step 1/11] Parsing and validating input arguments [ INFO ] Parsing input parameters [ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance,but it still may be non-optimal for some cases, for more information look at README.

[Step 2/11] Loading Inference Engine [ INFO ] InferenceEngine: API version ............ 2.1 Build .................. custom_releases/2019/R3_ac8584cb714a697a12f1f30b7a3b78a5b9ac5e05 Description ....... API [ INFO ] Device info: MYRIAD myriadPlugin version ......... 2.1 Build ........... 32974

[Step 3/11] Reading the Intermediate Representation network [ INFO ] Loading network files [ INFO ] Read network took 3.50 ms [Step 4/11] Resizing network to match image sizes and given batch [ INFO ] Network batch size: 200, precision: MIXED [Step 5/11] Configuring input of the model [Step 6/11] Setting device configuration [Step 7/11] Loading the model to the device [ INFO ] Load network took 1595.37 ms [Step 8/11] Setting optimal runtime parameters [Step 9/11] Creating infer requests and filling input blobs with images [ INFO ] Network input 'X' precision U8, dimensions (NCHW): 200 3 3072 1 [ WARNING ] No input files were given: all inputs will be filled with random values! [ INFO ] Infer Request 0 filling [ INFO ] Fill input 'X' with random values (image is expected) [ INFO ] Infer Request 1 filling [ INFO ] Fill input 'X' with random values (image is expected) [ INFO ] Infer Request 2 filling [ INFO ] Fill input 'X' with random values (image is expected) [ INFO ] Infer Request 3 filling [ INFO ] Fill input 'X' with random values (image is expected) [Step 10/11] Measuring performance (Start inference asyncronously, 4 inference requests, limits: 60000 ms duration) E: [xLink] [ 663113] [EventRead00Thr] eventReader:218 eventReader thread stopped (err -4) E: [xLink] [ 663113] [Scheduler00Thr] eventSchedulerRun:576 Dispatcher received NULL event! E: [global] [ 663113] [benchmark_app] XLinkReadDataWithTimeOut:1494 Event data is invalid E: [ncAPI] [ 663113] [benchmark_app] checkGraphMonitorResponse:1792 XLink error, rc: X_LINK_ERROR E: [ncAPI] [ 663113] [benchmark_app] ncGraphQueueInference:3979 Can't get trigger response E: [watchdog] [ 663788] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 664787] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 665786] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 666785] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 667784] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 668783] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 669782] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 670781] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 671781] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 672780] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 673779] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 674778] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 675777] [WatchdogThread] sendPingMessage:121 Failed send ping message: X_LINK_ERROR E: [watchdog] [ 675777] [WatchdogThread] watchdog_routine:315 [0x55ee881194e0] device, not respond, removing from watchdog

model_rectified.tar.gz

brmarkus commented 4 years ago

Still not working. I get similar error log messages like you (under Ubuntu 16.04, Myriad-X NCS2, OpenVINO v2020.1.002):

./benchmark_app -m FP16/inference_graph.xml -d MYRIAD
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance,but it still may be non-optimal for some cases, for more information look at README.

[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
        API version ............ 2.1
        Build .................. 37588
        Description ....... API
[ INFO ] Device info:
        MYRIAD
        myriadPlugin version ......... 2.1
        Build ........... 37588

[Step 3/11] Setting device configuration
[Step 4/11] Reading the Intermediate Representation network
[ INFO ] Loading network files
[ INFO ] Read network took 1.81 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 200, precision: MIXED
[Step 6/11] Configuring input of the model
[Step 7/11] Loading the model to the device
[ INFO ] Load network took 1668.54 ms
[Step 8/11] Setting optimal runtime parameters
[Step 9/11] Creating infer requests and filling input blobs with images
[ INFO ] Network input 'X' precision U8, dimensions (NCHW): 200 3 3072 1
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Infer Request 0 filling
[ INFO ] Fill input 'X' with random values (image is expected)
[ INFO ] Infer Request 1 filling
[ INFO ] Fill input 'X' with random values (image is expected)
[ INFO ] Infer Request 2 filling
[ INFO ] Fill input 'X' with random values (image is expected)
[ INFO ] Infer Request 3 filling
[ INFO ] Fill input 'X' with random values (image is expected)
[Step 10/11] Measuring performance (Start inference asyncronously, 4 inference requests, limits: 60000 ms duration)
E: [global] [    691478] [benchmark_app] XLinkReadData:165      addEventWithPerf(&event, &opTime) failed with error: 3
E: [ncAPI] [    691478] [benchmark_app] getGraphMonitorResponseValue:1887       XLink error, rc: X_LINK_COMMUNICATION_FAIL
E: [ncAPI] [    691478] [benchmark_app] ncGraphQueueInference:3427      Can't get trigger response
E: [global] [    691479] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691479] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691479] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691479] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691479] [benchmark_app] XLinkReadData:165      addEventWithPerf(&event, &opTime) failed with error: 3
E: [ncAPI] [    691479] [benchmark_app] getGraphMonitorResponseValue:1887       XLink error, rc: X_LINK_COMMUNICATION_FAIL
E: [ncAPI] [    691479] [benchmark_app] ncGraphQueueInference:3427      Can't get trigger response
E: [global] [    691479] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691480] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691480] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691480] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691480] [benchmark_app] XLinkReadData:165      addEventWithPerf(&event, &opTime) failed with error: 3
E: [ncAPI] [    691480] [benchmark_app] getGraphMonitorResponseValue:1887       XLink error, rc: X_LINK_COMMUNICATION_FAIL
E: [ncAPI] [    691480] [benchmark_app] ncGraphQueueInference:3427      Can't get trigger response
E: [global] [    691480] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691480] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691480] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691480] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691480] [benchmark_app] XLinkReadData:165      addEventWithPerf(&event, &opTime) failed with error: 3
E: [ncAPI] [    691480] [benchmark_app] getGraphMonitorResponseValue:1887       XLink error, rc: X_LINK_COMMUNICATION_FAIL
E: [ncAPI] [    691480] [benchmark_app] ncGraphQueueInference:3427      Can't get trigger response
F: [global] [    691482] [Scheduler00Thr] dispatcherLocalEventGetResponse:128   local NACK RTS. stream 'FIFO0' is full (event 56)

E: [global] [    691482] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691482] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691482] [benchmark_app] XLinkReadData:165      addEventWithPerf(&event, &opTime) failed with error: 3
E: [ncAPI] [    691482] [benchmark_app] getGraphMonitorResponseValue:1887       XLink error, rc: X_LINK_COMMUNICATION_FAIL
W: [ncAPI] [    691482] [benchmark_app] ncFifoDestroy:3132      myriad NACK

[Warning][VPU][MyriadPlugin] ncFifoDelete result NC_ERROR
E: [global] [    691482] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691482] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691482] [benchmark_app] XLinkReadData:165      addEventWithPerf(&event, &opTime) failed with error: 3
E: [ncAPI] [    691482] [benchmark_app] getGraphMonitorResponseValue:1887       XLink error, rc: X_LINK_COMMUNICATION_FAIL
W: [ncAPI] [    691482] [benchmark_app] ncFifoDestroy:3132      myriad NACK

[Warning][VPU][MyriadPlugin] ncFifoDelete result NC_ERROR
E: [global] [    691482] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_WRITE_REQ

E: [xLink] [    691482] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691482] [benchmark_app] XLinkReadData:165      addEventWithPerf(&event, &opTime) failed with error: 3
E: [ncAPI] [    691482] [benchmark_app] getGraphMonitorResponseValue:1887       XLink error, rc: X_LINK_COMMUNICATION_FAIL
[Warning][VPU][MyriadPlugin] Deallocate Graph result NC_ERROR.
W: [ncAPI] [    691482] [benchmark_app] ncDeviceClose:1724      Graphs on the device hasn't been destroyed! Graphs will be deallocated
W: [ncAPI] [    691482] [benchmark_app] ncDeviceClose:1732      Fifos on the device hasn't been destroyed! Fifos will be deallocated
E: [global] [    691482] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    691482] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691482] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    691482] [Scheduler00Thr] sendEvents:1036        Event sending failed
E: [global] [    691482] [Scheduler00Thr] dispatcherEventSend:53        Write failed (header) (err -4) | event XLINK_RESET_REQ

E: [xLink] [    691482] [Scheduler00Thr] sendEvents:1036        Event sending failed
[ ERROR ] Failed to queue inference: NC_ERROR
Yaffa16 commented 4 years ago

I am aslo getting the same error. I was notgetting this error till now and the model was working fine.

jgespino commented 2 years ago

Closing issue, if you are still need assistance please start a new issue and attach your model.