Frame marginalization strategy

[FengyuGuo]

currently we marginalize the oldest frame. but as in the code mentioned that a key frame selection can be done.

does it improve the accuracy or robustness if we did a key frame selection? have anybody tried?

  /**
   * @brief Will return the timestep that we will marginalize next.
   * As of right now, since we are using a sliding window, this is the oldest clone.
   * But if you wanted to do a keyframe system, you could selectively marginalize clones.
   * @return timestep of clone we will marginalize
   */
  double margtimestep() {
    double time = INFINITY;
    for (const auto &clone_imu : _clones_IMU) {
      if (clone_imu.first < time) {
        time = clone_imu.first;
      }
    }
    return time;
  }

[goldbattle]

The original MSCKF paper and follow ups by their groups have some discussions. A sliding window, shifting window, keyframing, or every third frame strategies can all give different advantages. In general, keyframing can improve the disparity of features during initialization and help initialize features even when stationary, but then you have to be careful to not re-use measurements multiple times which still can cause issues. Here is a paper that I believe has some discussion if I remember correctly: https://ieeexplore.ieee.org/abstract/document/6696807

[goldbattle]

There is also some discussion here on dynamic rates:

Zheng Huai and Guoquan Huang, Robocentric visual-inertial odometry, The International Journal of Robotics Research (IJRR), 2022 https://journals.sagepub.com/doi/10.1177/0278364919853361

Specifically:

It is important to note that this test is very challenging primarily due to: (i) several traffic lights where we must stop and wait for 15–25 seconds; (ii) frequent stop/yield signs before which we must stop or decelerate; (iii)dynamic scenes including running vehicles and pedestrians in the vicinity; (iv) strong lens flare when driving facing thesun; and (v) high speeds of vehicle when driving on some particular road sections (see Figure 9). .... For actual implementation, we employed a heuristic switching logic for the adaptive mode which is to be optimized in our future work; that is, the R-VIO changes the frequency of update from 15 to 7.5 Hz after detecting the speed descending (Dv < 0) for at least 2 seconds, and changes it back to 15 Hz once detecting the speed ascending (Dv > 0) for at least 2 seconds.

Example from results: