Closed joelbudu closed 1 year ago
@joelbudu , sorry for late reply. Would you please share more info about the model you used? Give me steps how to get it, then I can reproduce the issue.
Hello @LewisLiuPub, I am using the same mask rcnn model converted from the tensorflow object detection api on the COCO data set - mask_rcnn_inception_v2_coco_2018_01_28
Went through similar steps in the link below: https://github.com/intel/ros2_openvino_toolkit/tree/0.7.0_2019R3.1#launching
download and convert a trained model to produce an optimized Intermediate Representation (IR) of the model
cd /opt/openvino_toolkit/dldt/model-optimizer/install_prerequisites
sudo ./install_prerequisites.sh
mkdir -p ~/Downloads/models
cd ~/Downloads/models
wget http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
tar -zxvf mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
cd mask_rcnn_inception_v2_coco_2018_01_28
#FP32
sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --output_dir /opt/openvino_toolkit/models/segmentation/output/FP32
#FP16
sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --data_type=FP16 --output_dir /opt/openvino_toolkit/models/segmentation/output/FP16
I updated the .yaml file to point to the converted model and ran the Object Segmentation pipeline ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py
@joelbudu Recently we have upgraded this project to support new ROS2 version and OpenVINO version. In the new release v0.10.0, Mask RCNN was tested. Could you double check this feature with the new release?
Dear @LewisLiuPub . Is it possible to share some steps for testing the Mask RCNN. I'm not too clear on the procedure for v0.10.0.
Thanks
@joelbudu , ok, let me gather some info for you.
@LewisLiuPub any luck?
Hi, Currently, I can't find suitable environment to test this feature, but I hope the below steps help you:
NOTE: Below steps have been tested on Ubuntu 20.04.
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/src
git clone https://github.com/intel/ros2_openvino_toolkit -b galactic_dev
git clone https://github.com/intel/ros2_object_msgs
git clone https://github.com/IntelRealSense/realsense-ros.git -b ros2
git clone https://github.com/ros-perception/vision_opencv.git -b ros2
sudo apt-get install ros-galactic-diagnostic-updater
source /opt/ros/galactic/setup.bash
source /opt/intel/openvino_2021/bin/setupvars.sh
cd ~/catkin_ws
colcon build --symlink-install
source ./install/local_setup.bash
Preparation
source /opt/intel/openvino_2021/bin/setupvars.sh
sudo mkdir -p /opt/openvino_toolkit
sudo ln -s /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader /opt/openvino_toolkit/models
sudo chmod 777 -R /opt/openvino_toolkit/models
See all available models
cd /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader
sudo python3 downloader.py --print_all
Download the optimized Intermediate Representation (IR) of model (execute once), for example:
cd /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader
sudo python3 downloader.py --name semantic-segmentation-adas-0001 --output_dir /opt/openvino_toolkit/models/semantic-segmentation/output
run object segmentation sample code input from RealSenseCamera.
ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py
run object segmentation sample code input from Image.
ros2 launch dynamic_vino_sample pipeline_segmentation_image.launch.py
Dear @LewisLiuPub, The example you just outlined is for a semantic segmentation model (semantic-segmentation-adas-0001) which differs from instance segmentation I was referring to (Mask RCNN). I realised the ObjectSegmentation model in this repository was changed from one that takes Mask RCNN like models to one for semantic segmentation -> that was from version 0.9.0 .
Am I correct with my observation?
Kind regards, Joel
Hi, @joelbudu Yes, You are right, we have only tested the segmentation model listed in the Quick Start document. Let me know how urgent do you need MaskRCNN features, I am planning the development in 2022, but I am afraid not to make it ready as quickly as you want.
Dear @LewisLiuPub, given that this feature was already implemented previously in 0.8.0 2019R3.1 I assumed this was still going to be available in subsequent versions of this repository. I actually need this a bit urgently to be able to upgrade our OpenVINO version to anything greater than 2019. It's number one priority for us
I am also not clear on how it was replaced with a semantic segmentation model since the previous output in the pipeline was a people_msgs::msg::ObjectsInMasks.msg, is it using the same message or it is now using an Image.msg ?
Dear @joelbudu , could you please let me know which maskrcnn model you are using? I am a bit busy recently, but I would like to make a draft evaluation first.
@LewisLiuPub it's the one outlined in the documentation here: https://github.com/intel/ros2_openvino_toolkit/blob/master/doc/inferences/Object_Segmentation.md
dear @joelbudu , sorry for the late reply. Now we are preparing resource to fix this issue, appreciate if you can double confirm the model you are using is this one:
If so, you may have a quick check with the latest ROS2-OpenVINO code but reverting this patch . Please let me know whether if works or not.
Dear @LewisLiuPub . Yes I can confirm that the model I'm using is indeed the one linked above.
I will have to find some time to look into whether reverting the patch you've linked solves the issue. Thank you
@LewisLiuPub I tried reverting the patch however the build fails on my attempt with the following error:
--- stderr: dynamic_vino_lib
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/models/object_segmentation_model.cpp:34:50: error: no ‘void Models::ObjectSegmentationModel::setLayerProperty(InferenceEngine::CNNNetReader::Ptr)’ member function declared in class ‘Models::ObjectSegmentationModel’
InferenceEngine::CNNNetReader::Ptr net_reader)
^
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/models/object_segmentation_model.cpp:105:57: error: no ‘void Models::ObjectSegmentationModel::checkLayerProperty(const Ptr&)’ member function declared in class ‘Models::ObjectSegmentationModel’
const InferenceEngine::CNNNetReader::Ptr &net_reader)
^
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/models/object_segmentation_model.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /usr/include/x86_64-linux-gnu/c++/7/bits/c++allocator.h:33:0,
from /usr/include/c++/7/bits/allocator.h:46,
from /usr/include/c++/7/string:41,
from /usr/include/c++/7/bits/locale_classes.h:40,
from /usr/include/c++/7/bits/ios_base.h:41,
from /usr/include/c++/7/ios:42,
from /opt/ros/dashing/opt/yaml_cpp_vendor/include/yaml-cpp/parser.h:10,
from /opt/ros/dashing/opt/yaml_cpp_vendor/include/yaml-cpp/yaml.h:10,
from /home/rl001/ros2_ws/install/vino_param_lib/include/vino_param_lib/param_manager.hpp:22,
from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:20:
/usr/include/c++/7/ext/new_allocator.h: In instantiation of ‘void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel]’:
/usr/include/c++/7/bits/alloc_traits.h:475:4: required from ‘static void std::allocator_traits<std::allocator<_CharT> >::construct(std::allocator_traits<std::allocator<_CharT> >::allocator_type&, _Up*, _Args&& ...) [with _Up = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; std::allocator_traits<std::allocator<_CharT> >::allocator_type = std::allocator<Models::ObjectSegmentationModel>]’
/usr/include/c++/7/bits/shared_ptr_base.h:526:39: required from ‘std::_Sp_counted_ptr_inplace<_Tp, _Alloc, _Lp>::_Sp_counted_ptr_inplace(_Alloc, _Args&& ...) [with _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr_base.h:637:4: required from ‘std::__shared_count<_Lp>::__shared_count(std::_Sp_make_shared_tag, _Tp*, const _Alloc&, _Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr_base.h:1295:35: required from ‘std::__shared_ptr<_Tp, _Lp>::__shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr.h:344:64: required from ‘std::shared_ptr<_Tp>::shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel]’
/usr/include/c++/7/bits/shared_ptr.h:690:14: required from ‘std::shared_ptr<_Tp> std::allocate_shared(const _Alloc&, _Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}]’
/usr/include/c++/7/bits/shared_ptr.h:706:39: required from ‘std::shared_ptr<_Tp> std::make_shared(_Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}]’
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:322:79: required from here
/usr/include/c++/7/ext/new_allocator.h:136:4: error: invalid new-expression of abstract class type ‘Models::ObjectSegmentationModel’
{ ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/inferences/object_segmentation.hpp:28:0,
from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:51:
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/object_segmentation_model.hpp:28:7: note: because the following virtual functions are pure within ‘Models::ObjectSegmentationModel’:
class ObjectSegmentationModel : public BaseModel
^~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/vehicle_attribs_detection_model.hpp:22:0,
from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:33:
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/base_model.hpp:109:18: note: virtual bool Models::BaseModel::updateLayerProperty(InferenceEngine::CNNNetReader::Ptr)
virtual bool updateLayerProperty(InferenceEngine::CNNNetReader::Ptr network_reader) = 0;
^~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/pipeline_manager.cpp.o] Error 1
make[1]: *** [CMakeFiles/dynamic_vino_lib.dir/all] Error 2
make: *** [all] Error 2
---
Failed <<< dynamic_vino_lib [8.95s, exited with code 2]
I'm thinking there might be compatibility issues and others that need to be fixed after the patch.
Yes, you are right. there are some compatibility issues. @joelbudu Please let me know which ROS2 system (Dashing? Foxy? Galactic?) you are using.
I'm using ROS2 Dashing
On Jun 7, 2022 at 2:16 AM, <Lewis Liu @.***)> wrote:
Yes, you are right. there are some compatibility issues. @joelbudu (https://github.com/joelbudu) Please let me know which ROS2 system (Dashing? Foxy? Galactic?) you are using.
— Reply to this email directly, view it on GitHub (https://github.com/intel/ros2_openvino_toolkit/issues/211#issuecomment-1148090435), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAGY5PD6W7OFFVLSWFR5KBTVN2PIDANCNFSM5B57ZEGA). You are receiving this because you were mentioned.Message ID: @.***>
will close the issue. Mask RCNN instance segmentation has been supported in branch ros2 (supporting foxy/galactic).
Thank you for working on this @LewisLiuPub
Dear @LewisLiuPub, I am attempting to run a Mask RCNN Instance Segementation model on 0.9.0 2020.3 LTS, however I get the folowing error when running the pipeline_segmentation.launch.py sample with a custom trained Mask RCNN model.
Could you kindly assist?