Running Mask RCNN Instance Segmentation no longer working on 0.9.0 2020.3 LTS

joelbudu commented 3 years ago

Dear @LewisLiuPub, I am attempting to run a Mask RCNN Instance Segementation model on 0.9.0 2020.3 LTS, however I get the folowing error when running the pipeline_segmentation.launch.py sample with a custom trained Mask RCNN model.

Could you kindly assist?

[pipeline_with_params-1] [ DEBUG ] input size2
[pipeline_with_params-1] [ WARNING ] This inference sample should have only one input, but we got2inputs
[pipeline_with_params-1] [ INFO ] Segmentation model initialized.
[rviz2-2] [WARN] [rcl.logging_rosout]: Publisher already registered for provided node name. If this is due to multiple nodes with the same name then all logs for that logger name will go out over the existing publisher. As soon as any node with that name is destructed it will unregister the publisher, preventing any further logs for that name from being published on the rosout topic.
[pipeline_with_params-1] [ INFO ] Segmentation Engine initialized.
[pipeline_with_params-1] [ INFO ] Segmentation Inference instanced.
[pipeline_with_params-1] [ INFO ] Loading Network: Object Segmentation
[pipeline_with_params-1] [ INFO ]  ... Adding one Inference: ObjectSegmentation
[pipeline_with_params-1] [ INFO ] Updating connections ...
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[Image(1)<-->ObjectSegmentation(2)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[Image<-->ObjectSegmentation]
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[ObjectSegmentation(2)<-->ImageWindow(3)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[ObjectSegmentation<-->ImageWindow]
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[ObjectSegmentation(2)<-->RosTopic(3)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[ObjectSegmentation<-->RosTopic]
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[ObjectSegmentation(2)<-->RViz(3)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[ObjectSegmentation<-->RViz]
[pipeline_with_params-1] [ INFO ] One Pipeline Created!
[pipeline_with_params-1]  --> Image
[pipeline_with_params-1] Image --> ObjectSegmentation
[pipeline_with_params-1] ObjectSegmentation --> ImageWindow
[pipeline_with_params-1] ObjectSegmentation --> RosTopic
[pipeline_with_params-1] ObjectSegmentation --> RViz
[pipeline_with_params-1] [ DEBUG ] DEBUG: in Pipeline run process...
[pipeline_with_params-1] [ DEBUG ] DEBUG: Enqueue for detection: ObjectSegmentation
[pipeline_with_params-1] [ DEBUG ] first tensor2
[pipeline_with_params-1] [ DEBUG ] first tensor4
[pipeline_with_params-1] [ WARNING ] No input named: input
[pipeline_with_params-1] terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
[pipeline_with_params-1]   what():  [NOT_FOUND] Failed to find input or output with name: ''
[ERROR] [pipeline_with_params-1]: process has died [pid 20310, exit code -6, cmd '/home/rl001/ros2_ws_new/install/dynamic_vino_sample/lib/dynamic_vino_sample/pipeline_with_params -config /home/rl001/ros2_ws_new/install/dynamic_vino_sample/share/dynamic_vino_sample/param/pipeline_segmentation_image.yaml /openvino_toolkit/image_raw:=/camera/color/image_raw /openvino_toolkit/segmentation/segmented_obejcts:=/ros2_openvino_toolkit/segmented_obejcts /openvino_toolkit/segmentation/images:=/ros2_openvino_toolkit/image_rviz'].

LewisLiuPub commented 3 years ago

@joelbudu , sorry for late reply. Would you please share more info about the model you used? Give me steps how to get it, then I can reproduce the issue.

joelbudu commented 3 years ago

Hello @LewisLiuPub, I am using the same mask rcnn model converted from the tensorflow object detection api on the COCO data set - mask_rcnn_inception_v2_coco_2018_01_28

Went through similar steps in the link below: https://github.com/intel/ros2_openvino_toolkit/tree/0.7.0_2019R3.1#launching

download and convert a trained model to produce an optimized Intermediate Representation (IR) of the model

 cd /opt/openvino_toolkit/dldt/model-optimizer/install_prerequisites
 sudo ./install_prerequisites.sh
 mkdir -p ~/Downloads/models
 cd ~/Downloads/models
 wget http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
 tar -zxvf mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
 cd mask_rcnn_inception_v2_coco_2018_01_28
 #FP32
 sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --output_dir /opt/openvino_toolkit/models/segmentation/output/FP32
 #FP16
 sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --data_type=FP16 --output_dir /opt/openvino_toolkit/models/segmentation/output/FP16

I updated the .yaml file to point to the converted model and ran the Object Segmentation pipeline ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py

LewisLiuPub commented 2 years ago

@joelbudu Recently we have upgraded this project to support new ROS2 version and OpenVINO version. In the new release v0.10.0, Mask RCNN was tested. Could you double check this feature with the new release?

joelbudu commented 2 years ago

Dear @LewisLiuPub . Is it possible to share some steps for testing the Mask RCNN. I'm not too clear on the procedure for v0.10.0.

Thanks

LewisLiuPub commented 2 years ago

@joelbudu , ok, let me gather some info for you.

joelbudu commented 2 years ago

@LewisLiuPub any luck?

LewisLiuPub commented 2 years ago

Hi, Currently, I can't find suitable environment to test this feature, but I hope the below steps help you:

NOTE: Below steps have been tested on Ubuntu 20.04.

1. Environment Setup

Install ROS2 Galactic (guide)
Install Intel® OpenVINO™ Toolkit Version: 2021.4 (guide) or building by source code (guide)
Install Intel® RealSense ™ SDK (guide)

2. Building and Installation

Install ROS2_OpenVINO_Toolkit packages

mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/src
git clone https://github.com/intel/ros2_openvino_toolkit -b galactic_dev
git clone https://github.com/intel/ros2_object_msgs
git clone https://github.com/IntelRealSense/realsense-ros.git -b ros2
git clone https://github.com/ros-perception/vision_opencv.git -b ros2

Install dependencies

sudo apt-get install ros-galactic-diagnostic-updater

Build package

source /opt/ros/galactic/setup.bash 
source /opt/intel/openvino_2021/bin/setupvars.sh
cd ~/catkin_ws
colcon build --symlink-install
source ./install/local_setup.bash

3. Running the Demo

Preparation

source /opt/intel/openvino_2021/bin/setupvars.sh
sudo mkdir -p /opt/openvino_toolkit
sudo ln -s /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader /opt/openvino_toolkit/models
sudo chmod 777 -R /opt/openvino_toolkit/models

See all available models

cd /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader
sudo python3 downloader.py --print_all

Download the optimized Intermediate Representation (IR) of model (execute once), for example:

cd /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader
sudo python3 downloader.py --name semantic-segmentation-adas-0001 --output_dir /opt/openvino_toolkit/models/semantic-segmentation/output

run object segmentation sample code input from RealSenseCamera.

ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py

run object segmentation sample code input from Image.

ros2 launch dynamic_vino_sample pipeline_segmentation_image.launch.py

joelbudu commented 2 years ago

Dear @LewisLiuPub, The example you just outlined is for a semantic segmentation model (semantic-segmentation-adas-0001) which differs from instance segmentation I was referring to (Mask RCNN). I realised the ObjectSegmentation model in this repository was changed from one that takes Mask RCNN like models to one for semantic segmentation -> that was from version 0.9.0 .

Am I correct with my observation?

Kind regards, Joel

LewisLiuPub commented 2 years ago

Hi, @joelbudu Yes, You are right, we have only tested the segmentation model listed in the Quick Start document. Let me know how urgent do you need MaskRCNN features, I am planning the development in 2022, but I am afraid not to make it ready as quickly as you want.

joelbudu commented 2 years ago

Dear @LewisLiuPub, given that this feature was already implemented previously in 0.8.0 2019R3.1 I assumed this was still going to be available in subsequent versions of this repository. I actually need this a bit urgently to be able to upgrade our OpenVINO version to anything greater than 2019. It's number one priority for us

I am also not clear on how it was replaced with a semantic segmentation model since the previous output in the pipeline was a people_msgs::msg::ObjectsInMasks.msg, is it using the same message or it is now using an Image.msg ?

LewisLiuPub commented 2 years ago

Dear @joelbudu , could you please let me know which maskrcnn model you are using? I am a bit busy recently, but I would like to make a draft evaluation first.

joelbudu commented 2 years ago

@LewisLiuPub it's the one outlined in the documentation here: https://github.com/intel/ros2_openvino_toolkit/blob/master/doc/inferences/Object_Segmentation.md

LewisLiuPub commented 2 years ago

dear @joelbudu , sorry for the late reply. Now we are preparing resource to fix this issue, appreciate if you can double confirm the model you are using is this one:

If so, you may have a quick check with the latest ROS2-OpenVINO code but reverting this patch . Please let me know whether if works or not.

joelbudu commented 2 years ago

Dear @LewisLiuPub . Yes I can confirm that the model I'm using is indeed the one linked above.

I will have to find some time to look into whether reverting the patch you've linked solves the issue. Thank you

joelbudu commented 2 years ago

@LewisLiuPub I tried reverting the patch however the build fails on my attempt with the following error:

--- stderr: dynamic_vino_lib                                
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/models/object_segmentation_model.cpp:34:50: error: no ‘void Models::ObjectSegmentationModel::setLayerProperty(InferenceEngine::CNNNetReader::Ptr)’ member function declared in class ‘Models::ObjectSegmentationModel’
     InferenceEngine::CNNNetReader::Ptr net_reader)
                                                  ^
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/models/object_segmentation_model.cpp:105:57: error: no ‘void Models::ObjectSegmentationModel::checkLayerProperty(const Ptr&)’ member function declared in class ‘Models::ObjectSegmentationModel’
     const InferenceEngine::CNNNetReader::Ptr &net_reader)
                                                         ^
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/models/object_segmentation_model.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /usr/include/x86_64-linux-gnu/c++/7/bits/c++allocator.h:33:0,
                 from /usr/include/c++/7/bits/allocator.h:46,
                 from /usr/include/c++/7/string:41,
                 from /usr/include/c++/7/bits/locale_classes.h:40,
                 from /usr/include/c++/7/bits/ios_base.h:41,
                 from /usr/include/c++/7/ios:42,
                 from /opt/ros/dashing/opt/yaml_cpp_vendor/include/yaml-cpp/parser.h:10,
                 from /opt/ros/dashing/opt/yaml_cpp_vendor/include/yaml-cpp/yaml.h:10,
                 from /home/rl001/ros2_ws/install/vino_param_lib/include/vino_param_lib/param_manager.hpp:22,
                 from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:20:
/usr/include/c++/7/ext/new_allocator.h: In instantiation of ‘void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel]’:
/usr/include/c++/7/bits/alloc_traits.h:475:4:   required from ‘static void std::allocator_traits<std::allocator<_CharT> >::construct(std::allocator_traits<std::allocator<_CharT> >::allocator_type&, _Up*, _Args&& ...) [with _Up = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; std::allocator_traits<std::allocator<_CharT> >::allocator_type = std::allocator<Models::ObjectSegmentationModel>]’
/usr/include/c++/7/bits/shared_ptr_base.h:526:39:   required from ‘std::_Sp_counted_ptr_inplace<_Tp, _Alloc, _Lp>::_Sp_counted_ptr_inplace(_Alloc, _Args&& ...) [with _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr_base.h:637:4:   required from ‘std::__shared_count<_Lp>::__shared_count(std::_Sp_make_shared_tag, _Tp*, const _Alloc&, _Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr_base.h:1295:35:   required from ‘std::__shared_ptr<_Tp, _Lp>::__shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr.h:344:64:   required from ‘std::shared_ptr<_Tp>::shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel]’
/usr/include/c++/7/bits/shared_ptr.h:690:14:   required from ‘std::shared_ptr<_Tp> std::allocate_shared(const _Alloc&, _Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}]’
/usr/include/c++/7/bits/shared_ptr.h:706:39:   required from ‘std::shared_ptr<_Tp> std::make_shared(_Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}]’
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:322:79:   required from here
/usr/include/c++/7/ext/new_allocator.h:136:4: error: invalid new-expression of abstract class type ‘Models::ObjectSegmentationModel’
  { ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/inferences/object_segmentation.hpp:28:0,
                 from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:51:
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/object_segmentation_model.hpp:28:7: note:   because the following virtual functions are pure within ‘Models::ObjectSegmentationModel’:
 class ObjectSegmentationModel : public BaseModel
       ^~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/vehicle_attribs_detection_model.hpp:22:0,
                 from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:33:
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/base_model.hpp:109:18: note:     virtual bool Models::BaseModel::updateLayerProperty(InferenceEngine::CNNNetReader::Ptr)
     virtual bool updateLayerProperty(InferenceEngine::CNNNetReader::Ptr network_reader) = 0;
                  ^~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/pipeline_manager.cpp.o] Error 1
make[1]: *** [CMakeFiles/dynamic_vino_lib.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< dynamic_vino_lib [8.95s, exited with code 2]

I'm thinking there might be compatibility issues and others that need to be fixed after the patch.

LewisLiuPub commented 2 years ago

Yes, you are right. there are some compatibility issues. @joelbudu Please let me know which ROS2 system (Dashing? Foxy? Galactic?) you are using.

joelbudu commented 2 years ago

I'm using ROS2 Dashing

On Jun 7, 2022 at 2:16 AM, <Lewis Liu @.***)> wrote:

Yes, you are right. there are some compatibility issues. @joelbudu (https://github.com/joelbudu) Please let me know which ROS2 system (Dashing? Foxy? Galactic?) you are using.

— Reply to this email directly, view it on GitHub (https://github.com/intel/ros2_openvino_toolkit/issues/211#issuecomment-1148090435), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAGY5PD6W7OFFVLSWFR5KBTVN2PIDANCNFSM5B57ZEGA). You are receiving this because you were mentioned.Message ID: @.***>

LewisLiuPub commented 1 year ago

will close the issue. Mask RCNN instance segmentation has been supported in branch ros2 (supporting foxy/galactic).

joelbudu commented 1 year ago

Thank you for working on this @LewisLiuPub

intel / ros2_openvino_toolkit