Can I run SVO simply on stereo without IMU?

[lenardxu]

Currently I can run SVO on monocular sensing successfully? And I would further run it on stereo setup for comparison on my own dataset as done in your paper. However, it seems that all launching scripts related with stereo sensing are all performed with IMU (loosely coupled in frontend and tightly coupled). So, I would like to ask if this repo supports any implementation of runnning SVO simply based on stereo camera without IMU measurement? Thanks in advance.

[lenardxu]

I tried stereo setup without IMU information by doing the followings:

1. Build a new launch file based on the launch file "euroc_stereo_frontend_imu.launch", with launching commands related with IMU commented/removed, launching commands related with images subscibing corrected, and path to calibration file altered to the new one which corresponds to my camera
2. With regard to the newly built calibration file, it is based on the yaml file "euroc_stereo.yaml", where the intrinsics, distortion model and transformation matrices are all adapted to my case. To be noted, the transformation matrices for left and right camera are altered to the following:

   # For left camera
   T_B_C:
   cols: 4
   rows: 4
   data: [1.0, 0.0, 0.0, 0.0,
            0.0, 1.0, 0.0, 0.0,
            0.0, 0.0, 1.0, 0.0,
            0.0, 0.0, 0.0, 1.0]
   # For right camera
   T_B_C:
     cols: 4
     rows: 4
     data: [9.99723732e-01, -2.27311049e-02, 5.97663550e-03, -7.44190693e+00,
            2.26935912e-02, 9.99722779e-01, 6.27128733e-03, -1.43949956e-01,
            -6.11753203e-03, -6.13392377e-03, 9.99962449e-01, 7.51577914e-02,
            0.0, 0.0, 0.0, 1.0]

   As you can infer, the origin of body frame (IMU frame) is set to be coincided with the optical center of left camera so that the transformation matrix between left and right camera is registered in place of right camera.

With that done, when I first try run the new launch file, the following error takes place:

SUMMARY
========

CLEAR PARAMETERS
 * /svo/

PARAMETERS
 * /rosdistro: noetic
 * /rosversion: 1.15.14
 * /svo/T_world_imuinit/qw: 1
 * /svo/T_world_imuinit/qx: 0
 * /svo/T_world_imuinit/qy: 0
 * /svo/T_world_imuinit/qz: 0
 * /svo/calib_file: /home/rukangxu/MA...
 * /svo/cam0_topic: /stereo_publisher...
 * /svo/cam1_topic: /stereo_publisher...
 * /svo/depth_filter_affine_est_gain: True
 * /svo/depth_filter_affine_est_offset: True
 * /svo/detector_threshold_primary: 10
 * /svo/detector_threshold_secondary: 200
 * /svo/grid_size: 35
 * /svo/img_align_est_illumination_gain: True
 * /svo/img_align_est_illumination_offset: True
 * /svo/img_align_max_level: 4
 * /svo/img_align_min_level: 2
 * /svo/img_align_prior_lambda_rot: 0.0
 * /svo/img_align_prior_lambda_trans: 0.0
 * /svo/init_min_disparity: 30
 * /svo/kfselect_criterion: FORWARD
 * /svo/kfselect_min_angle: 6
 * /svo/kfselect_min_disparity: 40
 * /svo/kfselect_min_dist_metric: 0.001
 * /svo/kfselect_min_num_frames_between_kfs: 0
 * /svo/kfselect_numkfs_lower_thresh: 90
 * /svo/kfselect_numkfs_upper_thresh: 180
 * /svo/map_scale: 5.0
 * /svo/max_depth_inv: 0.029
 * /svo/max_fts: 180
 * /svo/max_n_kfs: 30
 * /svo/mean_depth_inv: 0.5
 * /svo/min_depth_inv: 2.857
 * /svo/n_pyr_levels: 3
 * /svo/pipeline_is_stereo: True
 * /svo/poseoptim_prior_lambda: 0.0
 * /svo/poseoptim_thresh: 2.0
 * /svo/poseoptim_using_unit_sphere: False
 * /svo/publish_every_nth_dense_input: 5
 * /svo/publish_marker_scale: 0.5
 * /svo/quality_max_drop_fts: 80
 * /svo/quality_min_fts: 40
 * /svo/reprojector_affine_est_gain: True
 * /svo/reprojector_affine_est_offset: True
 * /svo/reprojector_max_n_kfs: 5
 * /svo/scan_epi_unit_sphere: False
 * /svo/seed_convergence_sigma2_thresh: 200
 * /svo/update_seeds_with_old_keyframes: True
 * /svo/use_async_reprojectors: True
 * /svo/use_ceres_backend: False
 * /svo/use_imu: False
 * /svo/use_threaded_depthfilter: False

NODES
  /
    svo (svo_ros/svo_node)
    svo_gui (rqt_gui/rqt_gui)
    vis (rviz/rviz)

auto-starting new master
process[master]: started with pid [41957]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 9be08138-cfe3-11ec-973e-c9f871fab344
process[rosout-1]: started with pid [41970]
started core service [/rosout]
process[svo-2]: started with pid [41978]
process[vis-3]: started with pid [41979]
process[svo_gui-4]: started with pid [41980]
[ INFO] [1652133703.024027435]: Found parameter: pipeline_is_stereo, value: 1
[ WARN] [1652133703.025699570]: Cannot find value for parameter: set_initial_attitude_from_gravity, assigning default: 1
[ WARN] [1652133703.026175200]: Cannot find value for parameter: automatic_reinitialization, assigning default: 0
[ INFO] [1652133703.027465780]: Found parameter: calib_file, value: /home/rukangxu/MA_IAS/svo_ws/src/rpg_svo_pro_open/svo_ros/param/calib/oakd_stereo.yaml
F0510 00:01:43.028653 41978 rotation-quaternion-inl.h:115] Check failed: isValidRotationMatrix(matrix) 0000.999724 0-0.0227311 00.00597664
000.0226936 0000.999723 00.00627129
-0.00611753 -0.00613392 0000.999962
*** Check failure stack trace: ***
    @     0x7f6ee65bb703  google::LogMessage::Fail()
    @     0x7f6ee65c04ab  google::LogMessage::SendToLog()
    @     0x7f6ee65bb3ff  google::LogMessage::Flush()
    @     0x7f6ee65bbc2f  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f6ee6ac88b9  kindr::minimal::RotationQuaternionTemplate<>::RotationQuaternionTemplate()
    @     0x7f6ee6637582  YAML::convert<>::decode()
    @     0x7f6ee662052e  vk::cameras::NCamera::loadFromYaml()
    @     0x7f6ee72fef8b  svo::factory::loadCameraFromYaml()
    @     0x7f6ee730151c  svo::factory::makeStereo()
    @     0x7f6ee72d9235  svo::SvoInterface::SvoInterface()
    @     0x7f6ee7304651  svo_ros::SvoNodeBase::SvoNodeBase()
    @     0x561596562541  main
    @     0x7f6ee6eca0b3  __libc_start_main
    @     0x5615965625de  _start
[svo-2] process has died [pid 41978, exit code -6, cmd /home/rukangxu/MA_IAS/svo_ws/devel/lib/svo_ros/svo_node __name:=svo __log:=/home/rukangxu/.ros/log/9be08138-cfe3-11ec-973e-c9f871fab344/svo-2.log].
log file: /home/rukangxu/.ros/log/9be08138-cfe3-11ec-973e-c9f871fab344/svo-2*.log

It is about the rotation matrix checking. However, after checking the src file, I still have no clear clue what incurs that but guess that it's not allowed to set the transformatio matrices in the way shown above. So, would you tell me what that's about? Or is it simply not possible to run stereo setup without IMU?

[lenardxu]

To be continued from last comment -

After debugging, I found that problem can be traced back to the threshold controlling the validation on the rotation matrix, which is defined in the following code in the .h file "rotation-quaternion-inl.h" of https://github.com/ethz-asl/minkindr/blob/master/minkindr/include/kindr/minimal/implementation/

template <>
struct EPS<double> {
  static constexpr double value() { return 1.0e-8; }
  static constexpr double normalization_value() { return 1.0e-4; }
};

After checking my case, I increase that threshold by one magnitude, i.e., 1.0e-7, which I only guess should be acceptable in sense dealing with such a problem. However, despite of that problem being solved, running any accordingly adapted launch file related with stereo setup like "fla_stereo_imu.launch" or "euroc_stereo_frontend_imu.launch" can still not succeed, which reflects itself in no outcome, by checking the ros topics like "/svo/pose_cam/0", "/svo/pose_cam/1" and "/svo/pose_imu". Therefore, can anyone help me out of a jam? Many thanks in advance!

[lenardxu]

To be continued from last comment -

By further inspecting the code, the above problem should be attributed to the synchronization problem, i.e., the two image channels were not synchronized in my case when I fed them to the SVO stereo pipeline. As a solution, I wrote code for that purpose by means of "message_filters" based on approximate-time policy, which turned out to be effective given my observation that the SVO pipeline can subscribe to the image messages I fed. However, I still bumped into another problem, which was "Initialization failed. Not enough triangulated points." during run-time of the program. For that, I am first sure the manual initialization by moving the camera in cross fashion was followed. So does anyone have any idea about that problem taking place in stereo-setup processing? Thx again in advance.

[mohamedlaminetazir]

Any updates on that? I also intend to use SVO on stereo without IMU

[lenardxu]

Any updates on that? I also intend to use SVO on stereo without IMU

I have tried that setup yet, as I turned to vio. I may try again afterwards, out of interest. Have you managed it?

[MauUnterKarottenbaum]

ehhhh same issue

[lenardxu]

Any update about the issue where it is impossible to run the pipeline on the pure stereo camera?