is it possible to do Slam With one type of camera, and localization with another type?

[RashidLadj]

I want to do the SLAM phase with a 360 camera (THETA v), Images equirectangular, and the localization phase, with images taken by a smartphone, is it possible?

[laxnpander]

@RashidLadj: Interesting question. I am not a maintainer of OpenVSLAM and can only guess. In theory it might work. Every feature has a descriptor, which is basically a special way of representing the close environment of this feature. ORB uses a binary description in circular pattern I think. It just checks if the previous pixel is brighter than the current one (therefore either 0 / 1) and does this for a patch of pixels. It should work when description was done with one camera, and matching then with another. Stereo cameras are basically doing the same. However, in practice I think it will negatively influence your performance because false matches will increase. But you have to try how well it goes.

[RashidLadj]

@laxnpander I begin by thanking you for having taken the time to read my problem and answer it, Nonetheless, I did several tests and the localization does not work, if the descriptors are not of the same type of camera, it that's why I'm looking for a solution: /, my current idea is to try to convert a perspective image to an Equirectangular image with black on the edges, and try to do the Matching, friends I can't find any algorithm for do the conversion, I hope to have a clearer answer soon by the authors of OpenVSLAM

[YujiElfahkrany]

@RashidLadj why doesn't it work? I think we need more information to be able to identify the fix. I am not from the authors of openvslam but I am also interested in this approach. One thing I can think of is that the correct camera calibration was not performed. because if it was ORB is supposed to be scale and orientation invariant, pixel size from different lenses shouldn't matter.

[RashidLadj]

Hi @YujiElfahkrany , personally, I'm looking for a good explanation, by running openvslam (several times), on a map (cloud point) built with an equirectangular video, the algorithm could not match my perspective image and equirectangulars images , knowing that my equirectangular video is calibrated by default by Ricoh Theta software, and that my perspective image given as input for localization, is taken by a device from which I had recovered the calibration data, so my config.yaml file is correct, and I don't think the calibration data is very important for localization, in order to try to understand the problem, I took a feature point calculation and matching script in python on my side, and I tried to make the matching between my equirectangular image and my perspective image (my perspective image with several proportional resize), I almost never got a result, or at least almost never a matching for at least 4 points. So how do you do it? is there a certain resolution to take into account? ... @shinsumicco

[ymd-stella]

You need to be careful about image size and max_num_keypoints. In my case, I used the D435i to run SLAM in RGB-D and Localization in THETA S. A successful configuration is shown below. (I changed the parameters of the relocalizer a bit for performance improvement.)

#==============#
# Camera Model #
#==============#

Camera.name: "Realsense D435i"
Camera.setup: "RGBD"
Camera.model: "perspective"

Camera.fx: 458.83978271484375
Camera.fy: 458.865234375
Camera.cx: 325.91400146484375
Camera.cy: 189.83836364746094

Camera.k1: 0.0
Camera.k2: 0.0
Camera.p1: 0.0
Camera.p2: 0.0
Camera.k3: 0.0

Camera.fps: 30.0
Camera.cols: 640
Camera.rows: 360
Camera.focal_x_baseline: 22.942 # 5 cm

Camera.color_order: "BGR"

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 2600
Feature.scale_factor: 1.2
Feature.num_levels: 11
Feature.ini_fast_threshold: 20
Feature.min_fast_threshold: 7

#=====================#
# Tracking Parameters #
#=====================#

depth_threshold: 200.0 # depth_threshold * base_line[m] = 10m
depthmap_factor: 1000.0

#==============#
# Camera Model #
#==============#

Camera.name: "RICOH THETA S 960"
Camera.setup: "monocular"
Camera.model: "equirectangular"

Camera.fps: 15.0
Camera.cols: 1920
Camera.rows: 960
Camera.radius: 427.5

Camera.color_order: "RGB"

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 8000
Feature.scale_factor: 1.2
Feature.num_levels: 8
Feature.ini_fast_threshold: 25
Feature.min_fast_threshold: 12
Feature.mask_rectangles:
  - [0.0, 1.0, 0.0, 0.2]
  - [0.0, 1.0, 0.8, 1.0]

[YujiElfahkrany]

@ymd-stella what is Feature.mask_rectangles?

[YujiElfahkrany]

@ymd-stella successful means that you could localize with a different camera than that was used to build the map?

[ymd-stella]

Feature.mask_rectangles is a setting for Equirectangular cameras. It prevents the use of highly distorted areas.

successful means that you could localize with a different camera than that was used to build the map?

Yes. SLAM with D435i and localization with THETA S.

[YujiElfahkrany]

thank you, but doesnt it make more sense if we map by the wider lens and localize by the perspective? or am i missing something?

[ymd-stella]

but doesnt it make more sense if we map by the wider lens and localize by the perspective?

It depends on what you want to do. I wanted to localize with an equirectangular camera on a real scale map.

[YujiElfahkrany]

I see, thank you very much. I will try to map with 360 and localize with perspective.

[bot-lin]

Hi, @YujiElfahkrany Was your test going well?