Is the VI sensor data rectified or raw?

[zzhang001]

In https://daniilidis-group.github.io/penncosyvio/intrinsic_calib, it says the VI sensor calib data is based on rectified images, "The VI sensor calibration is based off the rectified images (this explains why the radial distortion is virtually zero), and is therefore the correct calibration to use for the rectified video frames in the ROS bag." but is the VI sensor data (I downloaded from http://visiondata.cis.upenn.edu/penncosyvio/tarfiles/visensor.tar) rectified or raw? We are using the calib based on rectified images and using the downloaded data but it seems wrong. There is a ROS bag for rectified images? Do you mind pointing me the link?

Thanks in advance.

[zzhang001]

@berndpfrommer thanks in advance!

[berndpfrommer]

For reasons of storage and to limit I/O, we (unfortunately!) only kept the rectified and compressed images. They are in the rosbag inside the tar file you downloaded (see output below). Did you use the frames or the mp4 files rather than the bag?

$ rosbag info seq_as.bag path: seq_as.bag version: 2.0 duration: 2:35s (155s) start: Apr 08 2016 14:37:39.48 (1460140659.48) end: Apr 08 2016 14:40:14.90 (1460140814.90) size: 267.8 MB messages: 149212 compression: none [347/347 chunks] types: sensor_msgs/CameraInfo [c9a58c1b0b154e0e6da7578cb991d214] sensor_msgs/CompressedImage [8f7a12909da2c9d3332d540a0977563f] sensor_msgs/Imu [6a62c6daae103f4ff57a132d6f95cec2] visensor_node/visensor_calibration [ed589a2fd70bfbada27d1d4266e0dea2] visensor_node/visensor_imu [2a6601ee1e26ed17ce49aea1dbd59920] visensor_node/visensor_time_host [d74fe546f2f8fbb3e0cfaff8b959c212] topics: /cam0/calibration 3109 msgs : visensor_node/visensor_calibration /cam0/camera_info 3109 msgs : sensor_msgs/CameraInfo
/cam1/calibration 3109 msgs : visensor_node/visensor_calibration /cam1/camera_info 3109 msgs : sensor_msgs/CameraInfo
/cust_imu0 31090 msgs : visensor_node/visensor_imu
/cust_mpu0 15540 msgs : visensor_node/visensor_imu
/cust_mpu1 15540 msgs : visensor_node/visensor_imu
/imu0 31090 msgs : sensor_msgs/Imu
/left/image_rect/compressed 3109 msgs : sensor_msgs/CompressedImage
/mpu0 15540 msgs : sensor_msgs/Imu
/mpu1 15540 msgs : sensor_msgs/Imu
/right/image_rect/compressed 3109 msgs : sensor_msgs/CompressedImage
/time_host 6218 msgs : visensor_node/visensor_time_host

[zzhang001]

Thanks. We "unzipped" the rosbag but the images are the same as the raw image files, so are the images rectified? We are using raw images.

We did some experiments on the raw images (we thought it's raw image), and between left/right, if we search along epipolar line and limit the search region +/- 2 pixels vertically, it can find very few correspondences. That's why we thought it's raw un-rectified images.

So basically you are saying both two types of images (in the as/left_cam/frames folder and the rosbag file) are all rectified?

[berndpfrommer]

Yes, the images are rectified. The VI sensor does that already in hardware. You can also see from the topic name, e.g. /left/image_rect/compressed. I should have some calibration bags that have april tags as well as the rectified images. There the corners of the tags should line up along the horizontal line between left and right image. I'm unfortunately very busy right now with other stuff, but will try to get to it within the next few days.

[berndpfrommer]

You are correct: the sensor was not properly calibrated and so the rectification is faulty! I did collect calibration images, and will try to get you the necessary transformation matrices before end of today. Very sorry about that!

[zzhang001]

That will be great if we can get either raw images + calib on raw images, or both rectified. Thanks! Really love to use your dataset in our paper.

[berndpfrommer]

The attached zip file has the results from a kalibration with Kalibr (camera-camera only, not imu). The calibration was run on the rectified images, i.e. the images that are in the bag of the data set. If you apply this calibration the rectified images should become truly rectified.

Unfortunately I did not have time yet to test the results with the calibration (feature detection + matching along horizontal line). Will try to do it this week. For now I hope the calib was successful (pdf file with report, looks reasonable).

visensor_calib.zip

[zzhang001]

Thanks, will give it a try. So I'll assume the left_T_imu extrinsic is correct, and use the updated left_T_right extrinsic to calculate the right_T_imu.

[zzhang001]

With the new calib (we trust imu_T_right extrinsics, and use the new right_T_left to get imu_T_left = imu_T_right * right_T_left), It seems correct (at least good enough) for our VIO system to run on AS dataset with VI sensor alone to get ~1m gap on ~150m trajectory. Thanks for the help! We would love to include your dataset and paper as reference in our to-be-submitted paper.

[berndpfrommer]

Glad to hear it worked out!

If you send me the trajectories for the BS and BF datasets I can tell you the errors for those as well.

Bernd

On Fri, Sep 1, 2017 at 2:40 AM, zzhang001 notifications@github.com wrote:

With the new calib (we trust imu_T_right extrinsics, and use the new right_T_left to get imu_T_left = imu_T_right * right_T_left), It seems correct (at least good enough) for our VIO system to run on AS dataset with VI sensor alone to get ~1m gap on ~150m trajectory. Thanks for the help! We would love to include your dataset and paper as reference in our to-be-submitted paper.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/daniilidis-group/penncosyvio/issues/2#issuecomment-326502852, or mute the thread https://github.com/notifications/unsubscribe-auth/AF61auv3X-UKm0h3QVw9YMvsgWIiDzSzks5sd6b6gaJpZM4O998- .