GoPro .bag playback [init] failure

[cod3monk3y]

I am playing back a .bag file which I've created using gopro_ros, and it's failing to init properly. I've fully calibrated following your brilliant estimation video, including the 3.5-hour Allan Variance IMU noise characterization.

The IMU data is ahead of the video by about 9 msec, which I've confirmed by hand (comparing frame-by-frame video in an NLE to IMU data using .bag import in PlotJuggler) and via Kalibr-ation. I believe my init failures are related to this time offset and the initial static init/jerk settings. I have a few questions related to this.

1. Where does Kalibr dynamic timeshift_cam_imu go?

During dynamic calibration, I get a 9 msec offset in cam0:

cam0:
  T_cam_imu:
  ...
  line_delay: 1.4475801213046952e-05
  resolution: [960, 540]
  timeshift_cam_imu: 0.009025714171251965

Where should this value go in OpenVINS? I tried putting it in kalibr_imu_chain:

# kalibr_imu_chain.yaml
imu0:
  ...
  #time_offset: -0.009025714171251965
  time_offset: 0.009025714171251965

and leaving it in cam0 when pasting into kalibr_imucam_chain.yaml

# kalibr_imucam_chain.yaml
cam0:
  ...
  timeshift_cam_imu: -0.009025714171251965

I also tried +/- variations in both files. None of these fix the init failure.

2. Could you explain these init_* settings, and confirm if they are set correctly?

In my test video, I have0.891 seconds with the camera still, then a movement that changes the magnitude of the acceleration vector to > 12.953. So I've set the following in estimator_config.yaml:

init_window_time:    0.75     # how many seconds to collect initialization information
init_imu_thresh:     2.0      # threshold for variance of the accelerometer to detect a "jerk" in motion
init_max_disparity: 10.0      # max disparity to consider the platform stationary (dependent on resolution)

I've got the init window time set to 0.75, which is before the camera moves for the first time at 0.891. The gravity magnitude is ~9.81 and the first camera movement has accelerometer magnitude of 12.953, which should exceed the IMU threshold (12.953 > 9.81+2 = 11.81) at 0.891sec.

I interpret init_max_disparity as (x, y) visual movement of features to indicate there is a change in the video/camera feed. I've left this as set in the example files.

Here's the plot of my IMU data

3. Could you clarify these init failure messages, please?

These are the three errors during init, in roughly this order:

[TIME]: 0.0021 seconds total (468.8 hz, 0.43 ms behind)
[init]: disparity is 0.073,0.075 (10.00 thresh)
[init]: failed static init: no accel jerk detected
...
[TIME]: 0.0024 seconds total (417.5 hz, 0.15 ms behind)
[init]: disparity is 0.083,16.452 (10.00 thresh)
[init-s]: no IMU excitation, below threshold 0.680 < 2.000
...
[TIME]: 0.0025 seconds total (394.0 hz, 0.48 ms behind)
[init]: disparity is 16.445,13.471 (10.00 thresh)
[init]: failed static init: no accel jerk detected, platform moving too much

My interpretation is:

• failed static init: no accel jerk detected - we have not yet detected IMU "jerk" movement greater than the threshold gravity_mag + init_imu_thresh which for me is 9.81 + 2 = 11.81
• no IMU excitation, below threshold 0.680 < 2.000 - there is visual movement from feature tracking as the disparity (16.452) is exceeding init_max_disparity (10.0), but the accelerometer is not moving. This seems related to the camera-imu timing offset not being set correctly.
• failed static init: no accel jerk detected, platform moving too much - we've passed the initialization time (0.75s), never detected the jerk motion, and the visual features are moving significantly. Init has failed and will not recover.

Are these interpretations correct? This all seems related to the timing offset not being set correctly. Could you recommend some next steps to investigate? E.g. does the video init time need to be longer, like 2 seconds?

Any help is appreciated. Sorry this is so lengthy.

[cod3monk3y]

I recorded a new video with 5 seconds of idle time, and changed my init configuration to

init_window_time: 5.5
init_imu_thresh:  1.5

and initialization succeeded!

I'm still curious about the settings, specifically the timing offset, as the position tracking "flies off" very quickly and inaccurately.

[goldbattle]

1) There isn't such a thing for imu time offset (would only be the case if there was multiple IMUs, but this code doesn't support that). So don't set time_offset in the imu.yaml. Both of these files should be direct copies of the output from Kalibr, no need to change them besides some of the indentation and adding the %YAML comment at the top.

2) These thresholds are the to detect if the platform is stationary / when it is picked up. We need to have a small stationary period at the beginning to initialize, and it is not possible to initialize when stationary, thus we want to find the moment the device is picked up.

init_window_time:    0.75     # how many seconds to collect initialization information
init_imu_thresh:     0.5      # threshold for variance of the accelerometer to detect a "jerk" in motion
init_max_disparity: 5.0      # max disparity to consider the platform stationary (dependent on resolution)

These will probably work for you based on the log you posted. A small window is better as it is just used to compute the average acceleration and its variance. The init_imu_thresh is basically how varied the accel readings are, so this can be tuned to how noisy your IMU is (when it is stationary what is this value vs the moment you pick it up). Just set the init_max_disparity to zero and see if the 0.5 detects when you pick it up.

The init_max_disparity is just there as a sanity check that we should only init once we get some disparity (e.g. have started moving). This will depend on your camera resolution, but for 640x480 somewhere around 2-5px should work well.

[cod3monk3y]

Brilliant. Thanks for the quick response! Here's a successful test run from the GoPro Hero9, walking 2 loops around my room.

Cheers, and great work on this project! /cm