SwinIR models and RealESRGAN_x2plus cannot be converted to NCNN

[theflyingzamboni]

I have attempted to convert these models using both PNNX and doing PTH -> ONNX -> NCNN (using convertmodel.com), and neither one works. This seems to be because of numerous operations that NCNN still does not support that are used in other frameworks like PyTorch and ONNX. Considering the usefulness of these models, it would be very helpful if they could be converted to NCNN. It would be beneficial if NCNN could be updated to support the various ops required by these models that are currently still unsupported, or at the least optimizations added to onnx2ncnn and pnnx to remove these ops and/or convert them to supported ops.

error log | 日志或报错信息 | ログ

This is a subset of the errors thrown by convertmodel.com when trying to convert SwinIR L (RealESRGAN_x2plus is missing similar ops):

Gather not supported yet!
# axis=0
Shape not supported yet!
Gather not supported yet!
# axis=0
Shape not supported yet!
Gather not supported yet!
# axis=0
Shape not supported yet!
Gather not supported yet!
# axis=0
Mod not supported yet!
# fmod=0
Mod not supported yet!
# fmod=0
Mod not supported yet!
# fmod=0
Mod not supported yet!
# fmod=0
Shape not supported yet!
Gather not supported yet!
# axis=0
Cast not supported yet!
# to=7
ConstantOfShape not supported yet!
# value 4
Unsupported slice step !
Cast not supported yet!
# to=7
Unknown data type 0
Shape not supported yet!
Gather not supported yet!
# axis=0
...
Cast not supported yet!
# to=7
Cast not supported yet!
# to=7
Cast not supported yet!
# to=7
Cast not supported yet!
# to=7
Unknown data type 0
Unknown data type 0
Unknown data type 0
...
Cast not supported yet!
# to=7
Cast not supported yet!
# to=7
Unknown data type 0
Unsupported transpose type !
Gather not supported yet!
# axis=0
Gather not supported yet!
# axis=0
Gather not supported yet!
# axis=0
Unknown data type 0
Unknown data type 0
Cast not supported yet!
# to=7
Cast not supported yet!
# to=7
Cast not supported yet!
# to=7
Cast not supported yet!
# to=7
Unknown data type 0
Unknown data type 0
Unknown data type 0
Erf not supported yet!

When attempting use PNNX, RealESRGAN_x2plus converts successfully to .pt using torch.jit.trace, but when trying to convert to NCNN using PNNX, no files are output even though no error is displayed. It fails silently.

When attempting to convert a SwinIR model using torch.jit.trace, I get these errors:

<input>:328: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
<input>:135: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
<input>:161: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
<input>:55: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
<input>:57: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
C:\Program Files\Python39\lib\site-packages\torch\jit\_trace.py:992: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Tensor-likes are not close!
Mismatched elements: 2407952 / 2408448 (100.0%)
Greatest absolute difference: 0.33713874220848083 at index (0, 0, 490, 454) (up to 1e-05 allowed)
Greatest relative difference: 0.6010049171521838 at index (0, 0, 297, 278) (up to 1e-05 allowed)
  _check_trace(

model | 模型 | モデル

1. 003_realSR_BSRGAN_DFOWMFC_s64w8_SwinIR-L_x4_GAN.pth
2. RealESRGAN_x2plus.pth

how to reproduce | 复现步骤 | 再現方法

1. Followed PNNX Readme instructions to convert .pth to .pt, and ran PNNX exe to try to convert to NCNN
2. Used torch.onnx.export to generate ONNX models, then attempted to convert to NCNN using convertmodel.com

[theflyingzamboni]

Any possibility of these ops being supported in the future?

[BahzBeih]

i can't convert "Janspiry/Image-Super-Resolution-via-Iterative-Refinement" model to NCNN too , i was using "chaiNNer" program, i hope they add support for this operations soon 🤞🤞🤞

[doms-17]

I can confirm, Real-ESRGAN_x2 model conversion to ncnn is not supported yet. I got the same error as above. There will ever be the chance it is going to be supported in the near future?

[redthing1]

Yeah, I see ERF missing :(

[nihui]

针对onnx模型转换的各种问题，推荐使用最新的pnnx工具转换到ncnn In view of various problems in onnx model conversion, it is recommended to use the latest pnnx tool to convert your model to ncnn

pip install pnnx
pnnx model.onnx inputshape=[1,3,224,224]

详细参考文档 Detailed reference documentation https://github.com/pnnx/pnnx https://github.com/Tencent/ncnn/wiki/use-ncnn-with-pytorch-or-onnx#how-to-use-pnnx