Understand how to handle Eigen::Ref conversions of yarp Images

[prashanthr05]

Within the ISensoBridge interface, we decided to pass image data using Eigen refs given the availability of suitable conversions to OpenCV that can be used by the client applications. (Please see https://github.com/dic-iit/bipedal-locomotion-framework/pull/87#issuecomment-682444518) for the discussion. In short, OpenCV offers to and fro conversions with Eigen Ref (https://github.com/opencv/opencv/blob/e3d502310f6ae0cad79f3474b9ea2b25705985f9/modules/core/include/opencv2/core/eigen.hpp).

Within the YARP implementations, we handle the images using yarp::sig::Image and YARP's Eigen helpers do not offer suitable Eigen ref conversions for Images, see https://www.yarp.it/namespaceyarp_1_1eigen.html.

(YARP offers OpenCV conversions if the usage of OpenCV is enabled. However, we want to circumvent the dependency of OpenCV in any case, so this is not a suitable option)

We need to understand these conversions to complete the missing implementation for getting images in YarpSensorBridge. https://github.com/dic-iit/bipedal-locomotion-framework/pull/106

[prashanthr05]

cc @traversaro @GiulioRomualdi @S-Dafarra would you have some pointers?

[S-Dafarra]

What would be a classic usage of these objects? What would you do once you have a yarp::sig::Image?

[prashanthr05]

I use yarp::sig::Image to read from different interfaces

• yarp::dev::IRGBDSensor which allows a read on yarp::sig::FlexImage and yarp::sig::ImageOf<yarp::sig::PixelFloat> for RGB and depth images respectively, for rgbd sensor clients
• yarp::dev::IFrameGrabber which allows a read on yarp::sig::ImageOf<yarp::sig::PixelRgb> for RGB images for frame grabber clients.

I do so by dynamic casting the yarp::sig::Image object which is a generic object.

Now, in order to pass this image to the user through the ISensorBridge, the user has to call getRGBImage or getDepthImage which accepts Eigen ref as an input argument.

In this case, I might have to do a necessary conversion from yarp::sig::Image to Eigen::ref. I wouldn't have to do any other manipulations.

What would you do once you have a yarp::sig::Image?

With the current implementation of the YarpSensorBridge, nothing but only to hold it in an internal buffer until a getXXX() is called, when the conversion described above must be done.

[S-Dafarra]

I see. I am not really familiar with those conversions. I guess that if the chain of conversions yarp->OpenCV->Eigen is possible, we can probably have yarp->Eigen without OpenCV. Have you tried to look at the implementations of these conversion methods?

[prashanthr05]

I see. I am not really familiar with those conversions. I guess that if the chain of conversions yarp->OpenCV->Eigen is possible, we can probably have yarp->Eigen without OpenCV. Have you tried to look at the implementations of these conversion methods?

I gave them a superficial look. I will try to report them here soon.

[traversaro]

There is something I am missing in general about using Eigen::Ref for images: In general, images are a rank-3 tensor, i.e. they have the width and depth, and then the number of channels, that for example for a RGB image are 3 (R, G and B), while for depth or grayscale images are 1. For this reason, if we want to support color images, an Eigen::Ref object may be not enough, and instead a Eigen::Tensor (and so an Eigen::TensorRef? ) may be more appropriate.

However, sorry for being a bit late for https://github.com/dic-iit/bipedal-locomotion-framework/pull/87#issuecomment-682444518, but I have a bit of a feeling that the design tradeoff for passing images around are quite different from the vector and matrices we are used in control. As an example, at least for the realsense the type of "channels" in the rgb image is not fixed, but can change at runtime https://github.com/robotology/yarp/blob/46c51b8872b984d2d587b20238d3e819dac1596e/src/devices/realsense2/realsense2Driver.cpp#L1010. An additional interesting observation that even in Drake, that as BLF tends to express as much as possible quantities using Eigen vectors, introduced a dedicate image class similar to the one used in YARP: https://drake.mit.edu/doxygen_cxx/classdrake_1_1systems_1_1sensors_1_1_image.html and https://github.com/RobotLocomotion/drake/blob/8e36e2e7bb17de3869432e77a07d39fb7a46adc6/systems/sensors/image.h .

I wonder if there is any specific reason for which we need to handle images and similar data structures using the ISensorBridge ? Could it make sense that for the time being we build a temporary "ExperimentalImageSensorBridge" that depends on any data structure is convenient a simple to use (YARP, OpenCV), and once we have a working prototype (and especially experience) of some algorithm that uses the images, we tried to migrate it to an improved ISensorBridge-interface that supports images?

[prashanthr05]

I wonder if there is any specific reason for which we need to handle images and similar data structures using the ISensorBridge ? Could it make sense that for the time being we build a temporary "ExperimentalImageSensorBridge" that depends on any data structure is convenient a simple to use (YARP, OpenCV), and once we have a working prototype (and especially experience) of some algorithm that uses the images, we tried to migrate it to an improved ISensorBridge-interface that supports images?

I think you made this point earlier. I had not understood it properly, or at least I thought we were talking about point clouds datatype only. But it does make sense to have a separate interface for images as you suggest. I shall refactor out handling the images out of the ISensorBridge.

About the Eigen ref over Tensor instead of MatrixXd, I think it makes sense. I will try to understand the use of Tensors and Eigen Refs over them. Anyhow, we might have to understand the conversion from YARP to Eigen. (The OpenCV conversions deals with Tensors).

[traversaro]

But it does make sense to have a separate interface for images as you suggest. I shall refactor out handling the images out of the ISensorBridge.

Note that it may be worth also to have some prototype code that does not use an interface at all (i.e. use directly the source of images) and once that works, the requirement for the interface will be more clear.