Using Airsim stereo camera image with stereo_image_proc (ros)

ryoma310 commented 2 years ago

Question

What's your question?

I try to use stereo_image_proc to depth estimation from images obtained by airsim_ros_pkg. It seems to work well as I see disparity image. But when I check the point cloud, it shows only one point.

Include context on what you are trying to achieve

Context details

This is my setting.json

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "Multirotor",
  "RecordUIVisible": true,
  "LogMessagesVisible": true,
  "ViewMode": "",
  "RpcEnabled": true,
  "PhysicsEngineName": "",
  "EnableCollisionPassthrogh": false,
  "SubWindows": [
    {"WindowID": 0, "CameraName": "front_left", "ImageType": 0, "VehicleName": "", "Visible": false, "External": false},
    {"WindowID": 1, "CameraName": "front_center", "ImageType": 0, "VehicleName": "", "Visible": false, "External": false},
    {"WindowID": 2, "CameraName": "front_right", "ImageType": 0, "VehicleName": "", "Visible": false, "External": false}
  ],
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 1936,
        "Height": 1296
      }
    ]
  },
  "SimMode": "Multirotor",
    "ClockType": "SteppableClock",
    "Vehicles": {
      "PX4": {
        "VehicleType": "PX4Multirotor",
        "AllowAPIAlways": true,
        "UseSerial": false,
        "LockStep": true,
        "UseTcp": true,
        "TcpPort": 4560,
        "ControlPortLocal": 14540,
        "ControlPortRemote": 14580,
        "Sensors":{
          "Barometer":{
              "SensorType": 1,
              "Enabled": true,
              "PressureFactorSigma": 0.0001825
          }
        },
        "Parameters": {
          "NAV_RCL_ACT": 0,
          "NAV_DLL_ACT": 0,
          "COM_OBL_ACT": 1,
          "COM_RCL_EXCEPT": 4,
          "LPE_LAT": 47.641468,
          "LPE_LON": -122.140165
        },
        "Cameras": {
          "front_center_custom": {
            "CaptureSettings": [
              {
                "PublishToRos": 1,
                "ImageType": 0,
                "FOV_Degrees": 27
              }
            ],
            "X": 0.50, "Y": 0, "Z": 0.10,
            "Pitch": 0, "Roll": 0, "Yaw": 0
          },
          "front_left_custom": {
            "CaptureSettings": [
              {
                "PublishToRos": 1,
                "ImageType": 0,
                "FOV_Degrees": 90,
                "Width": 1280,
                "Height": 720
              }
            ],
            "X": 0.50, "Y": -0.06, "Z": 0.10,
            "Pitch": 0.0, "Roll": 0.0, "Yaw": 0.0
          },
          "front_right_custom": {
            "CaptureSettings": [
              {
                "PublishToRos": 1,
                "ImageType": 0,
                "FOV_Degrees": 90,
                "Width": 1280,
                "Height": 720
              }
            ],
            "X": 0.50, "Y": 0.06, "Z": 0.10,
            "Pitch": 0.0, "Roll": 0.0, "Yaw": 0.0
          }
        },
        "X": 0, "Y": 0, "Z": 0,
        "Pitch": 0, "Roll": 0, "Yaw": 0
      }
    }
}

Airsim version: v1.8.1 - Linux
ROS: noetic

stereo_image_proc launch file


<launch>
<node name="transform" pkg="topic_tools" type="transform"
  args="/stereo/disparity /stereo/disparity_image sensor_msgs/Image 'm.image'" />

<!-- Launch stereo_image_proc node which runs OpenCV's block matching  -->
<node ns="stereo" pkg="stereo_image_proc" type="stereo_image_proc" name="stereo_image_proc">
<param name="stereo_algorithm" type="double" value="1.0" />
<param name="correlation_window_size" type="double" value="19.0" />
<param name="disparity_range" type="double" value="32.0" />
<param name="uniqueness_ratio" type="double" value="40.0" />
<param name="speckle_size" type="double" value="1000.0" />
<param name="speckle_range" type="double" value="2.0" />
<param name="approximate_sync" value="true" />
</node>

<node ns="/" pkg="topic_tools" type="relay" name="left_scene" args="/airsim_node/PX4/front_left_custom/Scene /stereo/left/image_raw" />
<node ns="/" pkg="topic_tools" type="relay" name="left_scene_info" args="/airsim_node/PX4/front_left_custom/Scene/camera_info /stereo/left/camera_info" />
<node ns="/" pkg="topic_tools" type="relay" name="right_scene" args="/airsim_node/PX4/front_right_custom/Scene /stereo/right/image_raw" />
<node ns="/" pkg="topic_tools" type="relay" name="right_scene_info" args="/airsim_node/PX4/front_right_custom/Scene/camera_info /stereo/right/camera_info" />

<!-- Launch static transform publishers --> 
<!--
<node pkg="tf" type="static_transform_publisher" name="tf_front_camera"
    args="0 0 0 -1.57 0 -1.57 fcu front_camera_link 10"/>
<node pkg="tf" type="static_transform_publisher" name="tf_right_camera"
    args="0 0 0 -3.14 0 -1.57 fcu right_camera_link 10"/>
<node pkg="tf" type="static_transform_publisher" name="tf_left_camera"
    args="0 0 0 0 0 -1.57 fcu left_camera_link 10"/> -->


---
- RVis
![Screenshot from 2022-07-21 16-03-46](https://user-images.githubusercontent.com/38186602/180151195-bb6ebf35-e55b-4f7a-861b-0a085716972b.png)

---
- camera info output
```log
$ rostopic echo /stereo/right/camera_info 
header: 
  seq: 531
  stamp: 
    secs: 1658384189
    nsecs: 637780428
  frame_id: "front_right_custom_optical"
height: 720
width: 1280
distortion_model: ''
D: []
K: [640.0, 0.0, 640.0, 0.0, 640.0, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [640.0, 0.0, 640.0, 0.0, 0.0, 640.0, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
$ rostopic echo /stereo/left/camera_info 
header: 
  seq: 536
  stamp: 
    secs: 1658384201
    nsecs: 632036448
  frame_id: "front_left_custom_optical"
height: 720
width: 1280
distortion_model: ''
D: []
K: [640.0, 0.0, 640.0, 0.0, 640.0, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [640.0, 0.0, 640.0, 0.0, 0.0, 640.0, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---

Include details of what you already did to find answers

I search about it and found that camera_info may have same error. I checked the topic detail and find that the projection matrix of left and right camera are same. (because of this, stereo_image_proc node cannot calculate the depth.)

If you have some idea, please let me know.

ryoma310 commented 2 years ago

I resolved it by myself!

CWC107753035 commented 2 years ago

For those who struggling on this question I find out the problem.

Because in airsim ros-node it doesn't provide baseline in camera_info. So if you want to get the point cloud through disparity image, you need to provide the Tx parameter manually in one of projection matrix in camera_info. see also sensor_msgs/CameraInfo.msg for more information. Cheers.

ryoma310 commented 2 years ago

@CWC107753035 That's exactly what I did to solve this issue. Thanks.

kanishkanarch commented 1 year ago

@ryoma310, how did you arm and takeoff this drone? I cannot see any service or topic to do so:

CWC107753035 commented 1 year ago

I think you can use ros service call to takeoff. Maybe /airsim_node /PX4/takeoff ?

Kanishk Vishwakarma @.***>於 2023年6月23日週五，15:32寫道：

@ryoma310 https://github.com/ryoma310, how did you arm and takeoff this drone? I cannot see any service or topic to do so: [image: image] https://user-images.githubusercontent.com/52484751/248202631-0cf0d0cb-c25e-42bf-b3d6-8c2b6f0778a0.png [image: image] https://user-images.githubusercontent.com/52484751/248202524-b9edf858-2c3a-4083-87db-1683e9249be8.png

— Reply to this email directly, view it on GitHub https://github.com/microsoft/AirSim/issues/4634#issuecomment-1603856853, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOHFC7R7YTRGWLMOICTKWI3XMVBCXANCNFSM54GP3TZQ . You are receiving this because you were mentioned.Message ID: @.***>

kanishkanarch commented 1 year ago

I am running a rosservice call /airsim_node/PX4/takeoff "waitOnLastTask: false" but drone is not taking off (not even arming).

kanishkanarch commented 1 year ago

There also seems to be no option to set the state (OFFBOARD/ALTITUDE/POSITION etc.)

CWC107753035 commented 1 year ago

According what you said, i think your mavlink doesn't connect to your airsim. Does your QGC can connect to your airsim? If so , maybe you can try to takeoff by using QGC GUI?

kanishkanarch commented 1 year ago

I am new to PX4, I did not setup PX4 sitl for AirSim. Now I did and it's working perfectly fine!! Thank you very much!

microsoft / AirSim