gazebosim / gazebo-classic

Gazebo classic. For the latest version, see https://github.com/gazebosim/gz-sim
http://classic.gazebosim.org/
Other
1.19k stars 481 forks source link

Support wide-angle and depth cameras in MultiCameraSensor #2760

Open scpeters opened 4 years ago

scpeters commented 4 years ago

It looks like the MultiCameraSensor does not support wide-angle cameras. Currently a single wide-angle camera can be created by using sdf <sensor type ='wideanglecamera'. That xml attribute is matched against the strings used in the GZ_REGISTER_STATIC_SENSOR macro (see WideAngleCameraSensor's macro for example). Currently, that sensor type attribute is the only way to indicate that a wideanglecamera sensor should be used (see example in lensflare_plugin.world). During WideAngleCameraSensor::Init, it calls Scene::CreateWideAngleCamera.

A MultiCameraSensor uses the "multicamera" type attribute string and creates sub-cameras based on the contents of multiple <camera> blocks within the <sensor> (see example in lensflare_plugin.world). During MultiCameraSensor::Init, it calls Scene::CreateCamera for each <camera> block that it finds. In order for MultiCameraSensor to contain wide-angle cameras, it would need some logic that allows it to call Scene::CreateWideAngleCamera, but we currently don't have a way to specify that, since the sensor type attribute is already set to "multicamera".

scpeters commented 4 years ago

Perhaps a //camera/@type attribute would be the right way to specify this to allow MultiCameraSensor to hold cameras of different types. It may take some refactoring of the gazebo sensor classes

kjeppesen1 commented 4 years ago

I added the camera type attribute to the camera.sdf file on the camera_type_attribute_6 branch on my fork. Brief testing seems to prove that this attribute is recognized when calling GetAttribute("type").

However, in discussion with @scpeters it seems that there are some larger architectural issues preventing the use of both wideanglecamera and depthcamera within the multicamera sensor. When adding an openni_kinect gazebo_ros plugin to the multicamera sensor, gazebo crashes and produces the following error:

Thread 1 "gzserver" received signal SIGSEGV, Segmentation fault. 0x00007fff57cc4e07 in gazebo::DepthCameraPlugin::Load(std::shared_ptr, std::shared_ptr) () from /usr/lib/x86_64-linux-gnu/gazebo-9/plugins/libDepthCameraPlugin.so (gdb)

My multi_cam_type branch on my gazebo fork demonstrates this issue when run with the included multi_camera.world. The error seems to be related to the mismatch between rendering::CameraPtr and rendering::DepthCameraPtr, as well as the sensor type not being "depth." Note that this code forces the creation of a depth camera regardless of the camera type, although some logic that creates the correct camera based on the type attribute is included as comments.

scpeters commented 4 years ago

Per our previous discussion and debugging session, @kjeppesen1 was attempting to use his fork of gazebo with a MultiCameraSensor modified to contain multiple depth cameras. He was also attempting to use the DepthCameraPlugin with these embedded depth sensors. There is a problem, though, because the DepthCameraPlugin expects to be attached to a DepthCameraSensor.