The compiled image_segmentation. CPP is used for inference without segmentation results

[wangshiwei567]

使用demo下的image_segmentation.cpp进行推理，没有分割结果 Run the int8 command: python tools/deploy.py /home/sw/PycharmProjects/mmdeploy/configs/mmseg/segmentation_tensorrt-int8_static-512x512.py
/home/sw/PycharmProjects/mmsegmentation-0.24.1/configs/pspnet/pspnet_r50-d8_512x512_20k_voc12aug.py /home/sw/PycharmProjects/mmsegmentation-0.24.1/train/psp/iter_2000.pth /home/sw/PycharmProjects/mmsegmentation-0.24.1/demo/8221.jpg --work-dir work_dirs/pspnet_zj_all_int8/ --device cuda --show --dump-info

After converting to int8, use C++ code for reasoning: ./image_segmentation cuda /home/sw/PycharmProjects/mmdeploy/work_dirs/pspnet_zj_all_int8/ /home/sw/PycharmProjects/mmsegmentation-0.24.1/demo/8221.jpg

The saved image does not have a mask area.Ask for help, thank you.

[lzhangzz]

Can you please check if the mask of output_tensorrt.jpg in model directory is correct?

[wangshiwei567]

Can you please check if the mask of output_tensorrt.jpg in model directory is correct?

The image mask that was converted to Tensorrt is correct. Which configuration file do I need to use to convert my trained model to Tensorrt? /home/sw/PycharmProjects/mmsegmentation-0.24.1/configs/pspnet/pspnet_r50-d8_512x512_20k_voc12aug.py？

[wangshiwei567]

No solution has been found yet.

[lvhan028]

Hi, @wangshiwei567 Could you please paste the environment information by running python tools/check_env.py? We'll try to reproduce it. Meanwhile, could you do us a favor to verify if image_segmentation works well with the models converted by segmentation_tensorrt-int8_static-512x512.py?

[wangshiwei567]

@lvhan028 Hi, This is the result.

2022-06-28 14:17:04,031 - mmdeploy - INFO - Environmental information 2022-06-28 14:17:04,130 - mmdeploy - INFO - sys.platform: linux 2022-06-28 14:17:04,130 - mmdeploy - INFO - Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0] 2022-06-28 14:17:04,130 - mmdeploy - INFO - CUDA available: True 2022-06-28 14:17:04,130 - mmdeploy - INFO - GPU 0: NVIDIA GeForce RTX 3090 2022-06-28 14:17:04,130 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda 2022-06-28 14:17:04,130 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.1, V11.1.74 2022-06-28 14:17:04,130 - mmdeploy - INFO - GCC: gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0 2022-06-28 14:17:04,130 - mmdeploy - INFO - PyTorch: 1.8.0+cu111 2022-06-28 14:17:04,130 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:

GCC 7.3 C++ Version: 201402 Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683) OpenMP 201511 (a.k.a. OpenMP 4.5) NNPACK is enabled CPU capability usage: AVX2 CUDA Runtime 11.1 NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86 CuDNN 8.0.5 Magma 2.5.2 Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 2022-06-28 14:17:04,130 - mmdeploy - INFO - TorchVision: 0.9.0 2022-06-28 14:17:04,130 - mmdeploy - INFO - OpenCV: 4.5.5 2022-06-28 14:17:04,130 - mmdeploy - INFO - MMCV: 1.5.2 2022-06-28 14:17:04,130 - mmdeploy - INFO - MMCV Compiler: GCC 7.3 2022-06-28 14:17:04,131 - mmdeploy - INFO - MMCV CUDA Compiler: 11.1 2022-06-28 14:17:04,131 - mmdeploy - INFO - MMDeploy: 0.5.0+ac0b52f 2022-06-28 14:17:04,131 - mmdeploy - INFO -

2022-06-28 14:17:04,131 - mmdeploy - INFO - Backend information 2022-06-28 14:17:04,442 - mmdeploy - INFO - onnxruntime: 1.8.1 ops_is_avaliable : True 2022-06-28 14:17:04,461 - mmdeploy - INFO - tensorrt: 8.2.3.0 ops_is_avaliable : True 2022-06-28 14:17:04,479 - mmdeploy - INFO - ncnn: None ops_is_avaliable : False 2022-06-28 14:17:04,480 - mmdeploy - INFO - pplnn_is_avaliable: False 2022-06-28 14:17:04,480 - mmdeploy - INFO - openvino_is_avaliable: False 2022-06-28 14:17:04,480 - mmdeploy - INFO -

2022-06-28 14:17:04,480 - mmdeploy - INFO - Codebase information 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmdet: 2.25.0 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmseg: 0.24.1 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmcls: 0.23.1 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmocr: None 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmedit: None 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmdet3d: None 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmpose: None 2022-06-28 14:17:04,482 - mmdeploy - INFO - mmrotate: None

The transformed model uses Python(demo/python/image segmentation.py) for reasoning and works fine. Reasoning with python is normal, reasoning with c++ is abnormal.

[lzhangzz]

My test result of PSPNet on Cityscapes using C++ SDK is correct. Can you share your model and the test image with us so we can figure out the what's wrong?

[wangshiwei567]

@lzhangzz Please give me your email account.

[lzhangzz]

Wait a minute. Can you check if num of classes is set properly in pipeline.json in the converted model folder? Similar to

{
    "type": "Task",
    "module": "mmseg",
    "name": "postprocess",
    "component": "ResizeMask",
    "params": {
        "type": "PSPHead",
        "in_channels": 2048,
        "in_index": 3,
        "channels": 512,
        "pool_scales": [
            1,
            2,
            3,
            6
        ],
        "dropout_ratio": 0.1,
        "num_classes": 19,    # <-------- HERE
        "norm_cfg": {
            "type": "SyncBN",
            "requires_grad": true
        },
        "align_corners": false,
        "loss_decode": {
            "type": "CrossEntropyLoss",
            "use_sigmoid": false,
            "loss_weight": 1.0
        }
    },
    "output": [
        "post_output"
    ],
    "input": [
        "prep_output",
        "infer_output"
    ]
}

The C++ demo is using the value to generate the palette while the Python demo just uses the value 256. If the value is wrong you are likely to get wrong visualization results.

[wangshiwei567]

The models I train only have one category: tracks. pipeline.json There are 21 classes.

{
    "pipeline": {
        "input": [
            "img"
        ],
        "output": [
            "post_output"
        ],
        "tasks": [
            {
                "type": "Task",
                "module": "Transform",
                "name": "Preprocess",
                "input": [
                    "img"
                ],
                "output": [
                    "prep_output"
                ],
                "transforms": [
                    {
                        "type": "LoadImageFromFile"
                    },
                    {
                        "type": "Resize",
                        "keep_ratio": false,
                        "size": [
                            512,
                            512
                        ]
                    },
                    {
                        "type": "Normalize",
                        "mean": [
                            123.675,
                            116.28,
                            103.53
                        ],
                        "std": [
                            58.395,
                            57.12,
                            57.375
                        ],
                        "to_rgb": true
                    },
                    {
                        "type": "ImageToTensor",
                        "keys": [
                            "img"
                        ]
                    },
                    {
                        "type": "Collect",
                        "keys": [
                            "img"
                        ],
                        "meta_keys": [
                            "img_norm_cfg",
                            "img_shape",
                            "filename",
                            "flip",
                            "pad_shape",
                            "ori_shape",
                            "ori_filename",
                            "scale_factor",
                            "flip_direction",
                            "valid_ratio"
                        ]
                    }
                ]
            },
            {
                "name": "psp",
                "type": "Task",
                "module": "Net",
                "input": [
                    "prep_output"
                ],
                "output": [
                    "infer_output"
                ],
                "input_map": {
                    "img": "input"
                }
            },
            {
                "type": "Task",
                "module": "mmseg",
                "name": "postprocess",
                "component": "ResizeMask",
                "params": {
                    "type": "PSPHead",
                    "in_channels": 2048,
                    "in_index": 3,
                    "channels": 512,
                    "pool_scales": [
                        1,
                        2,
                        3,
                        6
                    ],
                    "dropout_ratio": 0.1,
                    "num_classes": 21,
                    "norm_cfg": {
                        "type": "SyncBN",
                        "requires_grad": true
                    },
                    "align_corners": false,
                    "loss_decode": {
                        "type": "CrossEntropyLoss",
                        "use_sigmoid": false,
                        "loss_weight": 1.0
                    }
                },
                "output": [
                    "post_output"
                ],
                "input": [
                    "prep_output",
                    "infer_output"
                ]
            }
        ]
    }
}

[lvhan028]

@lzhangzz Please give me your email account.

lvhan@pjlab.org.cn zhangli@sensetime.com

[wangshiwei567]

@lvhan028 Hello, it has been sent, please check

[lvhan028]

@lvhan028 Hello, it has been sent, please check

I haven't received it yet. @lzhangzz how about you? @wangshiwei567 You may try to upload your model to this issue if it's size is not large.

[wangshiwei567]

@lvhan028 The file is too large to upload. I sent it again,please check.

[lzhangzz]

Sorry for the late reply, I just found your email in my spam folder.

In the ONNX model you provided, I found that the model do have 21 channels output before the ArgMax operator. Which means your model is actually trained with 21 classes.

On the other hand, the mask from the C API demo is also correct, the pixels of the track on image have label id 1. It's just the pseudo-randomly generated colors of labels 0 and 1 are too close to be distingushed.