ImageTransport in detectnet node

[r91andersson]

We're using an Nvidia Xavier as a client and a PC acting as a ROS master. The PC has two Realsense d435i cameras connected, and the Xavier is running the detectnet node and subscribing to the image streams coming from the PC.

The realsense are publishing the image streams via ImageTransport package, but I cant really see that this is done in the ros_compat.h. The way it subscribes to the image streams atm is by using ros::NodeHandle.

I will try to modify the code so that I will be using the ImageTransport package instead.

[WaldoPepper]

What advantage do you see in using image_transport in this node? E.g. the ROS node realsense-ros uses image_transport and publishes a raw image topic (which is the default transport for image_transport) sending raw sensor_msgs/Image which can be used by this detectnet node.

[r91andersson]

@WaldoPepper Yes, thats correct, but using raw-image topic and streaming this over a ROS network is not efficent. Instead every topic that subscribes to an image topic should do this with ImageTransport package.

To clarify, instead of detectnode is doing like this:

  ros::NodeHandle nh;
  ros::Subscriber sub = nh.subscribe("in_image_topic", 1, imageCallback);

It should be this:

ros::NodeHandle nh;
image_transport::ImageTransport it(nh);
image_transport::Subscriber sub = it.subscribe("in_image_base_topic", 1, imageCallback);

[WaldoPepper]

Interesting point. But if a node can ultimatly only process raw images, is there from a subscriber perspective still any advantage? IIRC image_transport does offer e.g. compressed formats the client need to be able to decode. Or can one benefit because the transmission is more efficient, even with raw images only?

[r91andersson]

That's a good question! I couldn't find any information about that. I will perform a test to find out that. I will use the method of subscribing to the raw stream as it is for now and measure the traffic on the ethernet, and compare to the second method to subscribe to raw image stream but via ImageTransport package. This would tell if there is any benefit of using ImageTransport from a subscriber perspective.

[r91andersson]

I can confirm that there is no change from a subscriber perspective of using either method 1 or method 2:

Method 1 ethernet payload without using ImageTransport to subscribe to raw image stream:

Method 2, ethernet payload with using ImageTransport to subscribe to raw image stream:

As you can see there is around 635 Mb/s streaming over the ethernet, and this is just for one image stream. Instead what I will do to decrease the payload on the ethernet is to subscribe to the compressed image stream and then decompress them at the Xavier to a raw image stream that can be handled with the detection node.

[r91andersson]

So my method 3: I created a new subscription method that subscribes to the compressed image stream in ros_compat.h. The ImageTransport can then decompress the image to the correct format.

#define ROS_CREATE_NODE(name)               \
        ros::init(argc, argv, name);        \
        ros::NodeHandle nh;             \
        ros::NodeHandle private_nh("~"); \
        image_transport::ImageTransport it(private_nh);

#define ROS_CREATE_COMPRESSED_IMAGE_TRANSPORT_SUBSCRIBER(msg, topic ,queue, callback)   it.subscribe(topic, queue, callback, ros::VoidPtr() ,image_transport::TransportHints("compressed"))

The result:

Compared to method 1 and 2 that had a payload of 650 Mb, the third method only adds around 66 Mb payload on the ethernet, allowing for multiple streams on the ROS network!

This would be a nice feature in the next update @dusty-nv ?

[WaldoPepper]

Very interesting insight, thanks for sharing!

From my point of view the approach you proposed would add a lot of value, as long as you keep the original raw mode as default and make the subscription to a compressed image topic selectable with an option (e.g. use_compressed). Switching to image_transport for all modes of transportation seems reasonable now.

[r91andersson]

I can't really see the benefit of using raw data to do the detection on. Because the model has been trained on compressed images (PNG,JPEG). So it would be a waste to run the raw image through the model.

But I can agree that it should tho be possible to run the raw image stream through detection.

[WaldoPepper]

That would be good, because not all nodes are capable of providing the compressed topic. Especially when you are using dusty's node as part of validation/test fixture in an CI/CD toolchain to deploy trained nets (which is what I am working on right now) there is a lot of raw data handling.

[r91andersson]

I see, yes, then it would be preferable to make TransportHints as an argument. With as you said, the default value resolving in raw_image.

[dusty-nv]

Because the model has been trained on compressed images (PNG,JPEG). So it would be a waste to run the raw image through the model.

Unless you are referring to a custom-trained model, the models are trained on raw RGB data, not compressed data. So the data would need uncompressed before feeding it into the DNN (which is fine)

[r91andersson]

Ok, good to know! Yes we're using a custom model that have been trained with jpeg images.