Datasets from mono and stereo cameras

[mslavescu]

Here is a dataset collected from two (not perfectly aligned) PS3 Eye cameras:

20180209_1736 - 2 x PS3 Eye + Android Sensors + OBD II data https://drive.google.com/open?id=1KBWIUBp5nZNDiNcKVJP1Tzqs_5ll5S4Y

Left camera (3 GB) https://drive.google.com/open?id=1WAv2m-APu0_rjMptRfK2DxRdvu2eWlvc Right camera (3 GB) https://drive.google.com/open?id=1-LMnWK2L_CRQMMBhGu9575HHhST2HLHP

Video with processed sample data from left camera (20180209_1736/left/001%4d.jpg): https://www.youtube.com/watch?v=_cEouvyJNMs

The dataset contains the images from left and right cameras, sensors data (GPS, Accelerometer, Gyro, Compass, Magnetometer from Android phone) collected with Android Sensors app and OBD II data collected with Torque Lite Android app.

The images needs to be aligned and synchronized (can be done by using the signal or break lights of the cars) also with the sensors and OBD II data (based on start/stop periods).

Both the alignment and synchronization should be done automatically, so a program to do this will be required, to ease the use of future datasets like this

[mslavescu]

We'll need some algorithms to align and synchronize the data (image/sensors) to maximize the accuracy of depth map and Visual SLAM (with IMU).

If you like to help please try to create a Google Colaboratory version also.

The dataset was collected between 17:36 and 18:06 EST, so it was almost dark and also was snowing lightly.

Lets see how robust are the stereo depth estimation, ego motion/speed estimation, and object detection/instance segmentation algorithms on this fairly difficult dataset

Would be interesting to use instance segmentation and then calculate disparity at contour level.

The images have noise (it was fairly dark), and the snow/wiper gets in the way too, so lets see if we could improve the accuracy of instance segmentation in these situations.

The main target would be to test #1-mvp-vbacc and create a 3D map (also correlating compass​, IMU, OBD II speed) to help #2-mvp-lkas (the lane markings are covered by snow).

[mslavescu]

For stereo (multi) camera depth map estimation, check this excellent thesis to also see how #fpga can help creating #3-mvp-smart-camera from monocular to multi camera setups, and what configurations are good for certain scenarios (number of cameras, resolution, fps) :

https://twitter.com/GTARobotics/status/875932414996094976

[ppirrip]

I dug a bit deeper on the topic / rosmsg type expected from the imu:

/Imu std_msgs/Header header uint32 seq time stamp string frame_id geometry_msgs/Quaternion orientation float64 x float64 y float64 z float64 w float64[9] orientation_covariance geometry_msgs/Vector3 angular_velocity float64 x float64 y float64 z float64[9] angular_velocity_covariance geometry_msgs/Vector3 linear_acceleration float64 x float64 y float64 z float64[9] linear_acceleration_covariance

It take a quaternion for orientation.

For fun I took a look at the node graph also:

The author recommends Kalibr for stereo calibration (another ros program expects a bag input

[mslavescu]

I'll try to build a camera like this and use it for datasets collection also:

https://github.com/harshatech2012/dual-raspi-stereo-vision

[mslavescu]

Lets try to run Kalibr in Colaboratory.

We just need to build a publisher with images/IMU, from the dataset, that is expected by these ROS calibration/disparity estimation tools.

[mslavescu]

We should try this also, from Skydio team:

End-to-End Learning of Geometry and Context for Deep Stereo Regression https://arxiv.org/abs/1703.04309

[mslavescu]

We should try and test it with this new approach also:

Dataset and Methods for Deep inertial Odometry http://deepio.cs.ox.ac.uk/