search

Tencent / ncnn

ncnn is a high-performance neural network inference framework optimized for the mobile platform
Other
20.58k stars 4.18k forks source link

PNNX can't convert yolov5s-seg.pt to ncnn #4488

Open Digital2Slave opened 1 year ago

Digital2Slave commented 1 year ago

error log | 日志或报错信息 | ログ

############# pass_level1
no attribute value
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
############# pass_level2
############# pass_level3
assign unique operator name pnnx_unique_0 to model.9.m
assign unique operator name pnnx_unique_1 to model.9.m
############# pass_level4
############# pass_level5
############# pass_ncnn
select along batch axis 0 is not supported
select along batch axis 0 is not supported
select along batch axis 0 is not supported
ignore Crop select_0 param dim=0
ignore Crop select_0 param index=0
ignore Crop select_1 param dim=0
ignore Crop select_1 param index=1
ignore Crop select_2 param dim=0
ignore Crop select_2 param index=2

model | 模型 | モデル

https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s-seg.pt

how to reproduce | 复现步骤 | 再現方法

  1. environment
Ubuntu 18.04.6 LTS
torch                     1.8.0+cpu
torchaudio                0.8.0
torchvision               0.9.0+cpu
cmake                     3.25.0
ninja                     1.11.1
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Build pnnx refer https://zhuanlan.zhihu.com/p/444022507.

  1. yolov5s-seg.pt

https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5s-seg.pt

  1. export.py

https://github.com/ultralytics/yolov5/blob/v7.0/export.py

  1. convert pt to torchscript
$ python export.py --weights /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.pt --include torchscript

export: data=data/coco128.yaml, weights=['/home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.pt'], imgsz=[640, 640], batch_size=1, device=cpu, half=False, inplace=False, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['torchscript']
YOLOv5 🚀 2022-11-22 Python-3.7.16 torch-1.8.0+cpu CPU

Fusing layers... 
YOLOv5s-seg summary: 224 layers, 7611485 parameters, 0 gradients, 26.4 GFLOPs

PyTorch: starting from /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.pt with output shape (1, 25200, 117) (14.9 MB)

TorchScript: starting export with torch 1.8.0+cpu...
TorchScript: export success ✅ 1.8s, saved as /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.torchscript (29.5 MB)

Export complete (4.0s)
Results saved to /home/tianzx/AI/pre_weights/yolov5-7.0/test
Detect:          python segment/detect.py --weights /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.torchscript 
Validate:        python segment/val.py --weights /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.torchscript 
PyTorch Hub:     model = torch.hub.load('ultralytics/yolov5', 'custom', '/home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.torchscript')  # WARNING ⚠️ SegmentationModel not yet supported for PyTorch Hub AutoShape inference
Visualize:       https://netron.app
  1. convert torchscript to ncnn
$ ./pnnx /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.torchscript inputshape=[1,3,640,640]
pnnxparam = /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.pnnx.param
pnnxbin = /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.pnnx.bin
pnnxpy = /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg_pnnx.py
pnnxonnx = /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.pnnx.onnx
ncnnparam = /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.ncnn.param
ncnnbin = /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg.ncnn.bin
ncnnpy = /home/tianzx/AI/pre_weights/yolov5-7.0/test/yolov5s-seg_ncnn.py
fp16 = 1
optlevel = 2
device = cpu
inputshape = [1,3,640,640]f32
inputshape2 = 
customop = 
moduleop = 
############# pass_level0
inline module = models.common.Bottleneck
inline module = models.common.C3
inline module = models.common.Concat
inline module = models.common.Conv
inline module = models.common.Proto
inline module = models.common.SPPF
inline module = models.yolo.Segment
inline module = models.common.Bottleneck
inline module = models.common.C3
inline module = models.common.Concat
inline module = models.common.Conv
inline module = models.common.Proto
inline module = models.common.SPPF
inline module = models.yolo.Segment

----------------

############# pass_level1
no attribute value
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
############# pass_level2
############# pass_level3
assign unique operator name pnnx_unique_0 to model.9.m
assign unique operator name pnnx_unique_1 to model.9.m
############# pass_level4
############# pass_level5
############# pass_ncnn
select along batch axis 0 is not supported
select along batch axis 0 is not supported
select along batch axis 0 is not supported
ignore Crop select_0 param dim=0
ignore Crop select_0 param index=0
ignore Crop select_1 param dim=0
ignore Crop select_1 param index=1
ignore Crop select_2 param dim=0
ignore Crop select_2 param index=2

The content of yolov5s-seg.ncnn.param file is not same content of @FeiGeChuanShu param file yolov5s-seg.param.

yolov5s-seg.ncnn.param

Could you help me out or figure out where is the problem ?

@nihui

Thanks a lot in advance any way!

nihui commented 1 year ago

https://github.com/Tencent/ncnn/pull/4533

Digital2Slave commented 1 year ago

4533

:+1: Great work! I will check it now.

Digital2Slave commented 1 year ago

4533

After update the ncnn repo codes, I use the rebuild pnnx tool to convert yolov5s-seg.torchscript model file, no attribute value and unknown Parameter value kind prim::Constant of TensorType are still appear in ############# pass_level1. :disappointed: 

$ cd ~/Github/ncnn/
$ git log --oneline
4fc977d7 (HEAD -> master, origin/master, origin/HEAD) fold scalar constant, write scalar attribute correctly (#4533)
...
$ cd tools/pnnx/
$ rm -rf build/
$ cmake -Bbuild -GNinja
$ cmake --build build --config Release
$ cd build/src/
$ ./pnnx /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.torchscript inputshape=[1,3,640,640]
pnnxparam = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pnnx.param
pnnxbin = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pnnx.bin
pnnxpy = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg_pnnx.py
pnnxonnx = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pnnx.onnx
ncnnparam = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.ncnn.param
ncnnbin = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.ncnn.bin
ncnnpy = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg_ncnn.py
fp16 = 1
optlevel = 2
device = cpu
inputshape = [1,3,640,640]f32
inputshape2 = 
customop = 
moduleop = 
############# pass_level0
inline module = models.common.Bottleneck
inline module = models.common.C3
inline module = models.common.Concat
inline module = models.common.Conv
inline module = models.common.Proto
inline module = models.common.SPPF
inline module = models.yolo.Segment
inline module = models.common.Bottleneck
inline module = models.common.C3
inline module = models.common.Concat
inline module = models.common.Conv
inline module = models.common.Proto
inline module = models.common.SPPF
inline module = models.yolo.Segment

----------------

############# pass_level1
no attribute value
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
############# pass_level2
############# pass_level3
############# pass_level4
############# pass_level5
############# pass_ncnn
nihui commented 1 year ago

no attribute value and unknown Parameter value kind prim::Constant of TensorType are infos not errors

nihui commented 1 year ago

build latest pnnx and ncnn python module

python export.py --weights yolov5s-seg.pt --include torchscript

saves yolov5s-seg.torchscript

./pnnx yolov5s-seg.torchscript inputshape=[1,3,640,640]

saves yolov5s-seg_pnnx.py and yolov5s-seg_ncnn.py etc.

import yolov5s-seg_pnnx
import yolov5s-seg_ncnn

yolov5s-seg_pnnx.test_inference()
yolov5s-seg_ncnn.test_inference()

gives you very close outputs, that means ncnn and pnnx results are on-par

Digital2Slave commented 1 year ago

build latest pnnx and ncnn python module

python export.py --weights yolov5s-seg.pt --include torchscript

saves yolov5s-seg.torchscript

./pnnx yolov5s-seg.torchscript inputshape=[1,3,640,640]

saves yolov5s-seg_pnnx.py and yolov5s-seg_ncnn.py etc.

import yolov5s-seg_pnnx
import yolov5s-seg_ncnn

yolov5s-seg_pnnx.test_inference()
yolov5s-seg_ncnn.test_inference()

gives you very close outputs, that means ncnn and pnnx results are on-par

Okay, I will check the converted ncnn model files later. 🤝

nihui commented 1 year ago

Hi I added a step by step tutorial on yolov5 ncnn deployment. pnnx tool is recommended for converting yolov5 model to ncnn, and reference code is provided. https://github.com/Tencent/ncnn/discussions/4541

Digital2Slave commented 1 year ago

Hi I added a step by step tutorial on yolov5 ncnn deployment. pnnx tool is recommended for converting yolov5 model to ncnn, and reference code is provided. https://github.com/Tencent/ncnn/discussions/4541

Thanks a lot, very appreciate it!

BTW，is it the similar way to handle the ncnn model based on yolov5s-seg.pt ?

Digital2Slave commented 1 year ago

build latest pnnx and ncnn python module

python export.py --weights yolov5s-seg.pt --include torchscript

saves yolov5s-seg.torchscript

./pnnx yolov5s-seg.torchscript inputshape=[1,3,640,640]

saves yolov5s-seg_pnnx.py and yolov5s-seg_ncnn.py etc.

import yolov5s-seg_pnnx
import yolov5s-seg_ncnn

yolov5s-seg_pnnx.test_inference()
yolov5s-seg_ncnn.test_inference()

gives you very close outputs, that means ncnn and pnnx results are on-par

Sorry for later response. I have an issue [1] 9618 segmentation fault (core dumped) python yolov5s-seg_ncnn.py when run ncnn model.

Reproduce steps

1. yolov5s-seg.pt -> yolov5s-seg.torchscript

~/Github/yolov5/yolov5-7.0 d2l ❯ python export.py --weights /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pt --include torchscript
export: data=data/coco128.yaml, weights=['/home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pt'], imgsz=[640, 640], batch_size=1, device=cpu, half=False, inplace=False, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['torchscript']
YOLOv5 🚀 2022-11-22 Python-3.7.16 torch-1.8.0+cpu CPU

Fusing layers... 
YOLOv5s-seg summary: 224 layers, 7611485 parameters, 0 gradients, 26.4 GFLOPs

PyTorch: starting from /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pt with output shape (1, 25200, 117) (14.9 MB)

TorchScript: starting export with torch 1.8.0+cpu...
TorchScript: export success ✅ 2.0s, saved as /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.torchscript (29.5 MB)

Export complete (3.8s)
Results saved to /home/tianzx/AI/pre_weights/test/yolov5-7.0
Detect:          python segment/detect.py --weights /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.torchscript 
Validate:        python segment/val.py --weights /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.torchscript 
PyTorch Hub:     model = torch.hub.load('ultralytics/yolov5', 'custom', '/home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.torchscript')  # WARNING ⚠️ SegmentationModel not yet supported for PyTorch Hub AutoShape inference
Visualize:       https://netron.app

2. yolov5s-seg.torchscript -> ncnn

As i use oh-my-zsh, so i add double quotes to inputshape.

~/Github/pnnx/pnnx-20230217-ubuntu d2l ❯ ./pnnx /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.torchscript "inputshape=[1,3,640,640]"
pnnxparam = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pnnx.param
pnnxbin = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pnnx.bin
pnnxpy = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg_pnnx.py
pnnxonnx = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.pnnx.onnx
ncnnparam = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.ncnn.param
ncnnbin = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.ncnn.bin
ncnnpy = /home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg_ncnn.py
fp16 = 1
optlevel = 2
device = cpu
inputshape = [1,3,640,640]f32
inputshape2 = 
customop = 
moduleop = 
############# pass_level0
inline module = models.common.Bottleneck
inline module = models.common.C3
inline module = models.common.Concat
inline module = models.common.Conv
inline module = models.common.Proto
inline module = models.common.SPPF
inline module = models.yolo.Segment
inline module = models.common.Bottleneck
inline module = models.common.C3
inline module = models.common.Concat
inline module = models.common.Conv
inline module = models.common.Proto
inline module = models.common.SPPF
inline module = models.yolo.Segment

----------------

############# pass_level1
no attribute value
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 5
no attribute value
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
unknown Parameter value kind prim::Constant of TensorType, t.dim = 1
############# pass_level2
############# pass_level3
############# pass_level4
############# pass_level5
############# pass_ncnn

3. Modify yolov5s-seg_ncnn.py

import numpy as np
import ncnn
import torch

def test_inference():
    torch.manual_seed(0)
    in0 = torch.rand(1, 3, 640, 640, dtype=torch.float)
    out = []

    with ncnn.Net() as net:
         net.load_param("/home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.ncnn.param")
         net.load_model("/home/tianzx/AI/pre_weights/test/yolov5-7.0/yolov5s-seg.ncnn.bin")

         with net.create_extractor() as ex:
            ex.input("in0", ncnn.Mat(in0.squeeze(0).numpy()).clone())

            _, out0 = ex.extract("out0")
            out.append(torch.from_numpy(np.array(out0)).unsqueeze(0))
            _, out1 = ex.extract("out1")
            out.append(torch.from_numpy(np.array(out1)).unsqueeze(0))

    if len(out) == 1:
        return out[0]
    else:
        return tuple(out)

if __name__ == '__main__':
    print(test_inference())

4. run yolov5s-seg_ncnn.py

~/AI/pre_weights/test/yolov5-7.0 48s d2l ❯ python yolov5s-seg_ncnn.py 
[1]    14508 segmentation fault (core dumped)  python yolov5s-seg_ncnn.py
glenn-jocher commented 1 year ago

@Digital2Slave I have great news 😃! I've recently added official support for Ultralytics YOLOv8 NCNN export ✅ in PR https://github.com/ultralytics/ultralytics/pull/3529 with the help of @nihui which is part of ultralytics==8.0.129. NCNN works for all tasks including Detect, Segment, Pose and Classify.

You can now export with CLI:

yolo export model=yolov8n.pt format=ncnn

or Python:

from ultralytics import YOLO

# Create a model
model = YOLO('yolov8n.pt')

# Export the model to NCNN with arguments
model.export(format='ncnn', half=True, imgsz=640)

Output is a yolov8n_ncnn_model/ directory containing model.bin, model.param and metadata.yaml, along with extra PNNX files. For details see https://github.com/pnnx/pnnx README.

To get this update:

Please let us know if NCNN export is working correctly for you, and don't hesitate to report any other issues you find or feature requests you may have. Happy training with YOLOv8 🚀!

Digital2Slave commented 1 year ago

@glenn-jocher Thanks a lot. I will check it later.

nihui commented 3 months ago

针对onnx模型转换的各种问题，推荐使用最新的pnnx工具转换到ncnn In view of various problems in onnx model conversion, it is recommended to use the latest pnnx tool to convert your model to ncnn

pip install pnnx
pnnx model.onnx inputshape=[1,3,224,224]

详细参考文档 Detailed reference documentation https://github.com/pnnx/pnnx https://github.com/Tencent/ncnn/wiki/use-ncnn-with-pytorch-or-onnx#how-to-use-pnnx