Support for compressed/prunded models

[alrzmshy]

Hello,

I can successfully run models trained with PyTorch (converted to ONNX) on NCS2. I have been trying to improve the FPS of models by pruning methods which is successfully done on the PyTorch model but the problem is that when I try to convert with torch.onnx.export, opset_version=10 it fails to convert to ONNX and with opset_version=11, conversion to ONNX is successful but while trying the benchmarkapp, I get : [ ERROR ] Failed to compile layer "829": unsupported layer type "ScatterNDUpdate".

Is there any future plans for supporting such layers? Are there any tricks to improve the FPS of the models on NCS2?

Thank you.

[jgespino]

Hi @alrzmshy

Which version of OpenVINO toolkit are you using? Did you build from source (which commit/branch)? There was a pull request for enabling ScatterND that has been merged. See https://github.com/openvinotoolkit/openvino/pull/1987.

I believe this operation is only implemented for CPU, are you able to run on CPU?

Regards, Jesus

[alrzmshy]

Hi @jgespino

I'm using the official 2021.2. In any case, if it only runs on CPU it won't be very useful for us.

[jgespino]

@alrzmshy Let me confirm with my peers if this is the case and get back to you.

[sevhabert]

@alrzmshy I confirm that we only support this layer on CPU for now.

[Flova]

I had a simmilar issue and solved it as descibed below. I followed the instructions in the OpenVINO docs and converted my PyTorch model to onnx. This resulted in the model having a ScatterND layer when I viewed the .onnx file in netron. But the model mainly consisted of normal 2d convolutions in the first place. Nothing fancy. After some expriments I found out that the ScatterND had been the result of the following formatiation during the output of the last layer:

x[..., 0:2] = (x[..., 0:2].sigmoid() + foo) * bar
x[..., 2:4] = torch.exp(x[..., 2:4])
x[..., 4:] = x[..., 4:].sigmoid()

When I changed it to

x = torch.cat([
    (x[..., 0:2].sigmoid() + foo) * bar,
    torch.exp(x[..., 2:4]),
    x[..., 4:].sigmoid(),
], axis=4)

the error was gone and it resulted in a much cleaner graph without ScatterND operations. So maybe try to avoid modifieing slices using this indexing pattern. It now works without further issues on my NCS2.

I also would recommend onnx-simplifier for a cleaner graph when debuing these issues.

[satpalsr]

Hey @jgespino, I am facing a similar issue while trying to convert XML to blob & I am on CPU. Failed to compile layer "ScatterND_735": unsupported layer type "ScatterNDUpdate"

I am using 2.1.2021.3.0-2787-60059f2c755-releases/2021/3 version

Used commands:

!source /opt/intel/openvino_2021/bin/setupvars.sh &&\
python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo.py --input_model latest.onnx --output_dir . --data_type FP32 --input_shape [1,3,512,512] &&\
/opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64/myriad_compile -ip U8 -VPU_NUMBER_OF_SHAVES 6 -VPU_NUMBER_OF_CMX_SLICES 6 -m latest.xml -o latest.blob

I tried onnx-simplifier but it didn't helped.

[jgespino]

Hi @satpalsr

Apologies for the delay, I did not see this response sooner as the issue was closed. The myriad_compile tool is for MYRIAD devices, ScatterND is only supported on CPU. Feel free to start a new issue if you have any additional questions.

[superkido511]

Hello,

I can successfully run models trained with PyTorch (converted to ONNX) on NCS2. I have been trying to improve the FPS of models by pruning methods which is successfully done on the PyTorch model but the problem is that when I try to convert with torch.onnx.export, opset_version=10 it fails to convert to ONNX and with opset_version=11, conversion to ONNX is successful but while trying the benchmarkapp, I get : [ ERROR ] Failed to compile layer "829": unsupported layer type "ScatterNDUpdate".

Is there any future plans for supporting such layers? Are there any tricks to improve the FPS of the models on NCS2?

Thank you.

Hello, could you share how did you prune the pytorch model? I'm trying to do the same on yolov7 models

[IkrameBeggar]

I had a simmilar issue and solved it as descibed below. I followed the instructions in the OpenVINO docs and converted my PyTorch model to onnx. This resulted in the model having a ScatterND layer when I viewed the .onnx file in netron. But the model mainly consisted of normal 2d convolutions in the first place. Nothing fancy. After some expriments I found out that the ScatterND had been the result of the following formatiation during the output of the last layer:

x[..., 0:2] = (x[..., 0:2].sigmoid() + foo) * bar
x[..., 2:4] = torch.exp(x[..., 2:4])
x[..., 4:] = x[..., 4:].sigmoid()

When I changed it to

x = torch.cat([
    (x[..., 0:2].sigmoid() + foo) * bar,
    torch.exp(x[..., 2:4]),
    x[..., 4:].sigmoid(),
], axis=4)

the error was gone and it resulted in a much cleaner graph without ScatterND operations. So maybe try to avoid modifieing slices using this indexing pattern. It now works without further issues on my NCS2.

I also would recommend onnx-simplifier for a cleaner graph when debuing these issues.

Hello, @Flova

I am facing the same issue, and I am a bit confused where I should apply this adjustment on my model. In addition, after modifying the code as you suggested, do I need to retrain the model?

I would really appreciate your help