search

luxonis / depthai-ros

Official ROS Driver for DepthAI Sensors.
MIT License
255 stars 186 forks source link

[Question] How to load a NN model from Model Zoo into depth_ai_ros driver ? #282

Open cwillboyd opened 1 year ago

cwillboyd commented 1 year ago

I am trying to load an NN model (fast_depth_480x640) that I downloaded from the model zoo. I have the .bin and .xml file on my local machine. What is the process of getting this to run using the camera.launch file in depthai_ros_driver ?

I believe I need to change a few lines in the camera.yaml file related to camera_i_nn_type: spatial and nn_i_nn_config_path: depthai_ros_driver/mobilenet.

Is it correct that I need to add a .json file in depthai_ros_driver/config/nn ? How do I convert the XML file from the model zoo into .json format?

Do I need to specify the model_path as well ? I see that there are some .blob files located in depthai-ros/depthai_examples/resources ... do I need to add a .blob file for my model here ? How can I convert the .bin and .xml files from the model zoo into .blob format ?

Lastly what should I set for camera_i_nn_type if I am using a nerual depth model based on RGB, should I change that value to rgb or keep it as spatial ?

Thank You

Serafadam commented 1 year ago

Hi, details on model conversion are available here, though currently to publish custom output from a NN you would either need to edit source code of depthai_ros_driver to include custom output or use depthai_bridge with custom callback function.

kgallowa commented 1 year ago

Working on the same type of problem (i.e. wanting to use a custom NN model with depthai-ros) and had the same question. @cwillboyd, did you end up figuring this out?

Serafadam commented 1 year ago

Hi @cwillboyd @kgallowa, corry for the late reply, in the newer versions of depthai_ros_driver you can load custom pipelines using ROS plugilib mechanism, that way you can create a custom NN node (similarly to how segmentation is set up) and load it into the pipeline.

wendwosenbb commented 4 months ago

@Serafadam is there away in the plugin mechanism to parse custom inference output? specifically, how to parse pre non_max_suppression results so that the host side handle the nms.

Serafadam commented 4 months ago

Hi, if you want to edit the node itself you would need to build the package from source. Alternatively, you can create your own wrapper as mentioned above, although for that currently you need to use dlopen (here you can find some more reference)