Incorporate filtering at a ROS node/driver level

[shaun-edwards]

I would like to incorporate my own filtering techniques on the sensor outputs (pointclouds and potentially 2D images). In order to achieve the best performance, this can be done in two ways:

• nodelets - eliminates costly message copying and sending, but requires the ROS node to be written in a certain way (which it currently is not)
• filter at a driver level - embed filtering at a driver level, but this brings up several questions:
  1. Can filters be applied at an image level (easier with opencv) that affect the point cloud output?
  2. Is there a generic way to do this? Are there a set of filters that we can agree should be supported in general? Here are my desired filters (in order they would be applied):
     • outlier removal - addresses flying/shadow points that occur around the edge
       • 2D options: opencv noise removal
       • 3D options: stastical outlier removal
     • erosion/dilation - fills holes and addresses pixels lost due to outlier removal
       • 2D options: erosiion/dilation
       • 3D options: erosion/dilation - maybe only available in v1.8.0
     • spatial averaging (average, median, bi-lateral) - smooths out surfaces similar to on-board filtering but "better" because of prior filtering.
       • 2D options: image filtering
       • 3D options: pointcloud filtering

There are more filtering options with 2D images. Ideally, these would be applied to the 2D images and then the pointcloud recalculated. However, as near as I can tell, the point cloud is calculated on the sensor and not in the ROS node.

[shaun-edwards]

@GarrettPlace

[tpanzarella]

Per your note above, I think the proper approach is to provide a nodelet interface and not have these specific filters (and their parameterizations) creep into the driver. To that end, I do not have plans to provide a nodelet interface in this library (PRs are welcome if you would like to do it). However, in the coming weeks ifm3d-ros will provide a nodelet interface.

As an aside, if you really need performance and are working with the O3D, you may want to consider running your code directly on the camera and using our low-latency framegrabber -- albeit, this will not be ROS.

Some other comments related to your post above...

spatial averaging (average, median, bi-lateral) ...

This can be achieved today, on the camera itself. You can specify it directly in the libo3d3xx JSON dump. The specific constants that map to the filter type can be found here

as near as I can tell, the point cloud is calculated on the sensor and not in the ROS node.

It depends. If your PCIC schema specifies it (which is the default for this ROS node), the Cartesian data will be computed on the camera and streamed over the wire. However, the point cloud data can always be reconstructed from the Unit Vectors (published on a latched topic), Radial Distance Image, and the Extrinsic Calibration. For concrete examples you can see this C++ example or even look at the ROS unit tests for a Python example.

[shaun-edwards]

@tpanzarella, thanks for the quick reply.

I wasn't aware of the ifm3d-ros project. Does it replace this project? Thanks for the pointer to the onboard library. I'll take a deeper look. The type of filtering I want to do could definitely be done on-board.

I know that spatial filtering can be done on board via config, but the noise around edges negatively effects the filtering. My approach is to remove the noise before applying the spatial filter. Because of this, I would have to apply the filter myself.

[tpanzarella]

I wasn't aware of the ifm3d-ros project. Does it replace this project?

Yes. ifm3d and ifm3d-ros will ultimately replace libo3d3xx and o3d3xx-ros. The ifm3d ecosystem is basically a generalization of the libo3d3xx ecosystem. For example, it currently supports both the O3D and O3X cameras from ifm.

Thanks for the pointer to the onboard library. I'll take a deeper look. The type of filtering I want to do could definitely be done on-board.

No problem. Yes, we run the filters you are talking about (and much more) directly on the O3D for various applications. However, you should know that the internal processor of the O3D is a bit constrained, so, running things like the PCL SOR filter won't be free.

[sara192]

Según su nota anterior, creo que el enfoque adecuado es proporcionar una interfaz de nodo y no tener estos filtros específicos (y sus parametrizaciones) en el controlador. Para ese fin, no tengo planes de proporcionar una interfaz de nodo en esta biblioteca (los RP son bienvenidos si desea hacerlo). Sin embargo, en las próximas semanas ifm3d-ros proporcionará una interfaz de nodo.

Por otro lado, si realmente necesita rendimiento y está trabajando con el O3D, puede considerar ejecutar su código directamente en la cámara y usar nuestro capturador de fotogramas de baja latencia , aunque esto no será ROS.

Algunos otros comentarios relacionados con su publicación anterior ...

Promedio espacial (promedio, mediana, bilateral) ...

This can be achieved today, on the camera itself. You can specify it directly in the libo3d3xx JSON dump. The specific constants that map to the filter type can be found here

as near as I can tell, the point cloud is calculated on the sensor and not in the ROS node.

It depends. If your PCIC schema specifies it (which is the default for this ROS node), the Cartesian data will be computed on the camera and streamed over the wire. However, the point cloud data can always be reconstructed from the Unit Vectors (published on a latched topic), Radial Distance Image, and the Extrinsic Calibration. For concrete examples you can see this C++ example or even look at the ROS unit tests for a Python example.

Hey

@GarrettPlace

Hey