create 3D point cloud out of ticino dataset

[surberj]

This process includes

• Choice of good footage (which images to choose)
• Feature extraction (use SIFT)
• Feature matching between images
• Bundle Adjustment (use ceres, pba, sba-levenberg-marquart)

Remark: Straightforward bundle adjustment estimates camera in- and extrinsics on the run and individually for all images. Since we use one camera, intrinsics have be the same for all images and can be calibrated and given as an input. Extrinsics are also known (with uncertainty) from the px4 ekf estimation.

[surberj]

VisualSFM is well suited for the feature extraction and feature matching (nice parallelized setup). For the bundle adjustment I will probably better use ceres, matlab or something else, since the parallel bundle adjustment, which is included in VisualSFM, is pretty limited (assumes first order distortion model, no straightforward option to include known camera extrinsics).

[surberj]

Some images of the first results:

• Reconstruction is based only on images and known intrinsics calibration, the extrinsics are not used
• Single parameter distortion model
• Using images of Take-off, Transition, first 100m of straight flight and one quarter-turn, a pretty consistent 3D model can be derived (Flat ground, forest below the plane can be seen)
• I used a lot of images for this reconstruction (all images (20hz) during transition and turns, a bit less (4hz) otherwise).

[surberj]

Issues:

• Points towards the horizon ( > 100m away) can not be reconstructed. Probably the triangulation fails as the baseline between the images used for the reconstruction is too small. Possible solution: Fly turns in both direction and use images from both turns (high baseline). Or work only with points nearby --> use down-looking camera in the future.
• The reconstruction fails in the first turn. At this moment we fly over the forest, which gives very poor correspondences. More useful features lie far away towards the horizon, which does not save use neither (see first issue). Working with known extrinsics could save us here, as "gaps" with too few correspondences do not harm us as much anymore.
• Scale: At this stage where we only use images and known intrinsics the scale can not be estiamted. As soon as we include known extrinsics this should be resolved.
• Verifying the consistency of the map will be a hard issue

[surberj]

Overview of pipeline images --> point cloud:

I do not see any out-of-the-box solution for doing the bundle adjustment with known camera extrinsics. I have to modify any BA optimization module. The easiest option would be probably to modify the pba (parallel bundle adjustment) code VisualSFM is working with. Other options would be to use directly the ceres-solver or do it in matlab.

[surberj]

It looks as if it is more or less straight forward with pba: Write a new load function in pba/src/pba/util.h to include camera poses from our PX4 EKF and include it into LoadModelFile(). But check first, how it will be used in the end: as a initial guess for the optimization or fixed or not at all ?

[surberj]

We decided to continue with maps created by OKVIS. For the challenging images of the ticino dataset the maps from OKVIS contain too much drift to be useful for re-localization (see midterm appendix). Therefore the ticino dataset was an important and interesting insight in the capabilities and limits of VI-odometry but won't play a major role anymore for my thesis.