run dm-vio with realsense d435i

jy84-choi commented 2 years ago

To run dm-vio with realsense d435i data, First, realsense d435i data was saved in EuRoC format. I also made camera.txt, imu.txt, and times.txt files And I ran dm-vio with the files. I'm getting the following nan error. When I took a log in the code, I checked that all residuals are in ResState::OOB. How can I solve this? Additionally, running with the --noimu option works well.

=================================================== 813-stats: 0 814-stats: 0 LOG 191: -nan fine. Res: 0 A, 0 L, 0 M; (0 / 0) forceDrop. a=0.000000, b=0.000000. Window 191 (2) GTData distance (seconds): 0.00414728 New ref frame id: 191 prepared keyframe id: 191 Prepared keyframe id: 191 lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! 0 0 0 0 0 0 AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! 0 0 0 0 0 0 AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! BIG ERROR! tracking failed entirely. Take predicted pose and hope we may somehow recover. Predicted pose: 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 Coarse Tracker tracked ab = 0.000000 0.000000 (exp 1.000000). Res nan! Time since last keyframe: 0.0333569 Frame history size: 193 Preparing keyframe: 192 Frames between KFs: 0 SPARSITY: MinActDist 0.600000 (need 1000 points, have 0 points)! OPTIMIZE 0 pts, 0 active res, 0 lin res! activeResiduals.size(): 0 lastEnergy: 0 0 0 lastEnergyL: 0 lastEnergyM: nan Initial Error A(0.000000)=(AV -nan). Num: A(0) + M(0); ab 0.000000 0.000000! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 0 (L -5.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 1 (L -3.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 2 (L -1.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 3 (L 1.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 4 (L 3.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 5 (L 5.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 6 (L 7.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 7 (L 9.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 8 (L 11.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 9 (L 13.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 10 (L 15.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 11 (L 17.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 12 (L 19.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 13 (L 21.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Dynamic weight: 1 STEPS: A -nan; B -nan; R nan; T -nan. REJECT 14 (L 23.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab -nan -nan! Num BA Iterations done: 15 LOG 192: -nan fine. Res: 0 A, 0 L, 0 M; (0 / 0) forceDrop. a=0.000000, b=0.000000. Window 192 (3) GTPose: -0.023315 -0.050298 0 GTData distance (seconds): 0.00231806 New ref frame id: 192 prepared keyframe id: 192 Prepared keyframe id: 192 lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! nan nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 1 with initOption 1 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! nan nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 2 with initOption 2 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 3 with initOption 3 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! 0 0 0 0 0 0 AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! 0 0 0 0 0 0 AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 4 with initOption 4 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 5 with initOption 5 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 6 with initOption 6 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 7 with initOption 7 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 8 with initOption 8 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 9 with initOption 9 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 10 with initOption 10 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 11 with initOption 11 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 12 with initOption 12 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 13 with initOption 13 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 14 with initOption 14 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 15 with initOption 15 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 16 with initOption 16 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 17 with initOption 17 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 18 with initOption 18 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 19 with initOption 19 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 20 with initOption 20 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 21 with initOption 21 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 22 with initOption 22 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 23 with initOption 23 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 24 with initOption 24 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 25 with initOption 25 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 26 with initOption 26 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 27 with initOption 27 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 28 with initOption 28 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 29 with initOption 29 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan lvl3, it -1 (l=0.010000 / 1.000000) INITIA: 0.000->-nan (0 -> 0) (|inc| = 0.000000)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) lvl 3, it 0 (l=0.010000 / 1.000000) REJECT: -nan->-nan (0 -> 0) (|inc| = -nan)! -nan -nan nan -nan -nan -nan AFF 0 0 (rel 1 0) inc too small, break! WARNING: Coarse tracker thinks that tracking was not good! RE-TRACK ATTEMPT 30 with initOption 30 and start-lvl 3 (ab 0.000000 0.000000): nan nan nan nan nan -> nan nan nan -nan nan BIG ERROR! tracking failed entirely. Take predicted pose and hope we may somehow recover. Predicted pose: -nan -nan -nan nan -nan -nan -nan -nan -nan -nan -nan -nan 0 0 0 1 TRACKING FAILED ENTIRELY, NO HOPE TO RECOVER TRACKING FAILED ENTIRELY, NO HOPE TO RECOVER X Error of failed request: BadDrawable (invalid Pixmap or Window parameter) Major opcode of failed request: 14 (X_GetGeometry) Resource id in failed request: 0x4400003 Serial number of failed request: 3278 Current serial number in output stream: 3278

lukasvst commented 2 years ago

Thanks for trying the noimu mode first. Have you also passed the camchain file and is it read correctly?

Could you please post the command you use for running and the beginning of the commandline output where all the settings are printed, until the error first occurs?

jy84-choi commented 2 years ago

Thank you for your reply!

I made camchain file using the default camera-inertial calibration values of realsense d435i.

My run command is below: python3 run_dmvio.py --output=console --dataset=sr --dmvio_settings=sr.yaml --withgui --iter=1 --only_seq=1 --dmvio_args="init_pgba_skipFirstKFs=1 init_requestFullResetNormalizedErrorThreshold=0.8"

Here, sr is my dataset name.

Since "#-stats: 0" repeats a lot, I deleted "#-stats:" log for readability.

----------- create_dmvio_commands ----------- files=./data calib=./camera.txt imuCalib=/home/jychoi/git_ws/src/dm-vio/configs/sr_calib/camchain.yaml mode=1 preset=0 nogui=0 useimu=1 quiet=0 settingsFile=/home/jychoi/git_ws/src/dm-vio/configs/sr.yaml init_pgba_skipFirstKFs=1 init_requestFullResetNormalizedErrorThreshold=0.8 WARNING: No end times for dataset. ----------- Save Project Status ----------- {'name': 'dmvioresult', 'dataset': 'sr', 'build_type': 'RelWithDebInfo', 'num_iter': 1, 'only_seq': 1, 'results_name': 'dmvioresult-sr-2022-05-17--09-57-11', 'config_name': 'workpc', 'date_run': datetime.datetime(2022, 5, 17, 9, 57, 11, 382691), 'realtime': False, 'temporary': False, 'noimu': False, 'quiet': False, 'output_type': 'console', 'withgui': True, 'custom_dmvio_args': 'init_pgba_skipFirstKFs=1 init_requestFullResetNormalizedErrorThreshold=0.8', 'dmvio_settings': 'sr.yaml', 'gdb': False, 'git_hash': '5bf2a67c10973a91c061483b10a0257e2e0c5b11', 'commit_message': 'Added note on the GTSAM version to the README.', 'commit_time': datetime.datetime(2022, 5, 4, 15, 26, 58), 'diff_empty': False, 'eval_tool_command': 'run_dmvio.py --output=console --dataset=sr --dmvio_settings=sr.yaml --withgui --iter=1 --only_seq=1 --dmvio_args=init_pgba_skipFirstKFs=1 init_requestFullResetNormalizedErrorThreshold=0.8', 'eval_tools_git_hash': 'f1e581e26022706b97aceee8022489e8a97e5c82', 'eval_tools_commit_message': 'Initial commit.', 'eval_tools_diff_empty': False, 'command_0': '/home/jychoi/git_ws/src/dm-vio/cmake-build-relwithdebinfo/bin/dmvio_dataset files=./data calib=./camera.txt imuCalib=/home/jychoi/git_ws/src/dm-vio/configs/sr_calib/camchain.yaml mode=1 preset=0 nogui=0 useimu=1 quiet=0 settingsFile=/home/jychoi/git_ws/src/dm-vio/configs/sr.yaml init_pgba_skipFirstKFs=1 init_requestFullResetNormalizedErrorThreshold=0.8 resultsPrefix=/home/jychoi/git_ws/src/dm-vio-result/dmvioresult-sr-2022-05-17--09-57-11/sr_220426_162017_cam_40_light_on_ir_color_2_0/ start=1', 'cwd0': '/home/jychoi/dataset/SR_VIO_EuRoC/220426_162017_cam_40_light_on_ir_color_2/cam0'}

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

----------- STARTING EXECUTION! ----------- Working Dir: /home/jychoi/dataset/SR_VIO_EuRoC/220426_162017_cam_40_light_on_ir_color_2/cam0 Command: /home/jychoi/git_ws/src/dm-vio/cmake-build-relwithdebinfo/bin/dmvio_dataset files=./data calib=./camera.txt imuCalib=/home/jychoi/git_ws/src/dm-vio/configs/sr_calib/camchain.yaml mode=1 preset=0 nogui=0 useimu=1 quiet=0 settingsFile=/home/jychoi/git_ws/src/dm-vio/configs/sr.yaml init_pgba_skipFirstKFs=1 init_requestFullResetNormalizedErrorThreshold=0.8 resultsPrefix=/home/jychoi/git_ws/src/dm-vio-result/dmvioresult-sr-2022-05-17--09-57-11/sr_220426_162017_cam_40_light_on_ir_color_2_0/ start=1 loading data from ./data! loading calibration from ./camera.txt! Loading imu parameters from /home/jychoi/git_ws/src/dm-vio/configs/sr_calib/camchain.yaml! Loading IMU parameter file at: /home/jychoi/git_ws/src/dm-vio/configs/sr_calib/camchain.yaml Used T_cam_imu: 1 0 0 0.006 0 1 0 -0.0051 0 0 1 -0.012 0 0 0 1 Used noise values: 0.00447213 0.0014142 0.316227 0.1 PHOTOMETRIC MODE WITHOUT CALIBRATION!

=============== PRESET Settings: =============== DEFAULT settings:

no real-time enforcing
2000 active points
5-7 active frames
1-6 LM iteration each KF
original image resolution

Enabling IMU integration! Loading settings from yaml file: /home/jychoi/git_ws/src/dm-vio/configs/sr_calib/camchain.yaml! START AT 1! Settings: accelerometer_noise_density: 0.2 accelerometer_random_walk: 0.04 addVisualToCoarseGraphIfTrackingBad: 0 alwaysCanBreakIMU: 0 baToCoarseAccBiasVariance: 1000 baToCoarseGyrBiasVariance: 0.05 baToCoarsePoseVariance: 0.1 baToCoarseRotVariance: 1 baToCoarseVelVariance: 0.1 dynamicWeightRMSEThresh: 8 fixKeyframeDuringCoarseTracking: 1 generalScaleIntervalSize: 60 gravityDirectionFixZ: 1 gyroscope_noise_density: 0.02 gyroscope_random_walk: 0.004 init_coarseScaleUncertaintyThresh: 1 init_conversionType: 0 init_disableVIOUntilFirstInit: 1 init_fixPoses: 1 init_initDSOParams: 1 init_lambdaLowerBound: 1e-16 init_maxNumPoses: 100 init_multipleBiases: 0 init_multithreadedInitDespiteNonRT: 0 init_onlyKFs: 1 init_pgba_conversionType: 0 init_pgba_delay: 100 init_pgba_prepareGraphAddDelValues: 0 init_pgba_prepareGraphAddFactors: 0 init_pgba_priorExtrinsicsRot: 0.01 init_pgba_priorExtrinsicsTrans: 0.1 init_pgba_priorGravityDirection: 0.4 init_pgba_priorGravityDirectionZ: 0.0001 init_pgba_reinitScaleUncertaintyThresh: 0.5 init_pgba_scaleUncertaintyThresh: 1 init_pgba_skipFirstKFs: 1 init_priorExtrinsicsRot: 0.01 init_priorExtrinsicsTrans: 0.1 init_priorGravityDirection: 0.4 init_priorGravityDirectionZ: 0.0001 init_priorRotSigma: 1e-05 init_priorTransSigma: 1e-05 init_requestFullResetErrorThreshold: -1 init_requestFullResetNormalizedErrorThreshold: 0.8 init_scalePriorAfterInit: 0 init_secondthreshGravdir: 1000 init_secondthreshScale: 1e+07 init_threshGravdir: 1000 init_threshScale: 1.02 init_transitionModel: 2 init_updatePoses: 1 integration_sigma: 0.316227 maxFrameEnergyThreshold: 5000 maxPreloadImages: 0 maxTimeBetweenInitFrames: 100000 numMeasurementsGravityInit: 40 preload: 0 priorExtrinsicsRot: 0.01 priorExtrinsicsTrans: 0.1 priorGravityDirection: 0.4 priorGravityDirectionZ: 0.0001 resultsPrefix: /home/jychoi/git_ws/src/dm-vio-result/dmvioresult-sr-2022-05-17--09-57-11/sr_220426_162017_cam_40_light_on_ir_color_2_0/ setting_forceNoKFTranslationThresh: 0 setting_maxOptIterations: 6 setting_minFramesBetweenKeyframes: -0.5 setting_minIdepth: 0.02 setting_minOptIterations: 1 setting_optGravity: 1 setting_optIMUExtrinsics: 0 setting_optScaleBA: 1 setting_prior_bias: 0 setting_prior_velocity: 0 setting_scaleFixTH: 0 setting_solverMode: 2048 setting_transferCovToCoarse: 1 setting_visualOnlyAfterScaleFixing: 0 setting_weightDSOCoarse: 0.001 setting_weightDSOToGTSAM: 1.66667e-05 setting_weightZeroPriorDSOInitX: 0 setting_weightZeroPriorDSOInitY: 0 skipFirstKeyframe: 0 transferCovToCoarseMultiplier: 1 updateDynamicWeightDuringOptimization: 1 use16Bit: 0 useScaleDiagonalHack: 0 Reading Calibration from file ./camera.txt ... found! found RadTan (OpenCV) camera model, building rectifier. Creating RadTan undistorter Input resolution: 640 480 In: 385.755000 385.755000 323.121000 236.743000 0.000000 0.000000 0.000000 0.000000 Out: No Rectification Output resolution: 640 480

Rectified Kamera Matrix: 385.755 0 323.121 0 385.755 236.743 0 0 1

NO PHOTOMETRIC Calibration! Reading Photometric Calibration from file PhotometricUndistorter: Could not open file! image num: 3013time num: 3013 set EXPOSURES to zero! got 3013 images and 3013 timestamps and 0 exposures.! ImageFolderReader: got 3013 files in ./data! IMU Id: 1650957618591295744 IMU-data too old -> skipping frame IMU-data too old -> skipping frame

====This sentence "IMU-data too old -> skipping frame" came up a lot. ====

IMU-data too old -> skipping frame Found no start frame for IMU-data! using pyramid levels 0 to 3. coarsest resolution: 80 x 60! START PANGOLIN! Using setting_minFramesBetweenKeyframes=0.5 because of non-realtime mode. Switching to initializer state: CoarseIMUInit PixelSelector: Using block sizes: 16, 16 Loading gt data GTData distance (seconds): 0.0078123 PixelSelector: Using block sizes: 16, 16 GTData distance (seconds): 0.0067623 InitTimeBetweenFrames: 0.0333583

==== GTdata distance and InitTimeBetweenFrames are repeated.====

Scaling with rescaleFactor: 1.28913 Initialization: keep 10.9% (need 1000, have 9202)! INITIALIZE FROM INITIALIZER (974 pts)! Frame history size: 192 Preparing keyframe: 191 Frames between KFs: 190 SPARSITY: MinActDist 1.900000 (need 1000 points, have 974 points)! OPTIMIZE 974 pts, 974 active res, 0 lin res! activeResiduals.size(): 974 Initial Error A(0.000000)=(AV -nan). Num: A(0) + M(0); ab 0.000000 0.000000! Dynamic weight: 1 STEPS: A nan; B nan; R nan; T nan. REJECT 0 (L -5.00, dir nan, ss 1.0): A(0.000000)=(AV -nan). Num: A(0) + M(0); ab nan nan! Dynamic weight: 1

lukasvst commented 2 years ago

The problem is that the IMU data cannot be associated with the image data. The line IMU-data too old -> skipping frame should only printed a couple of times maximum. If it is printed out a lot and with the line Found no start frame for IMU-data! there is no frame with associated IMU data.

Have you interpolated the IMU file, so that each frame has an IMU measurement with exactly its timestamp (e.g. with this script?

If that's not the issue, the association code makes some strong assumptions:

The first column in times.txt must be the timestamp of the corresponding image in nanoseconds. There should be a measurement in the IMU file with exactly the same timestamp.
Images are associated with the line in times.txt simply by order. It's best practice to name each image after the first column in times.txt (the timestamp in nanoseconds), but this is not checked at runtime.

I hope this helps in finding the issue, (otherwise you could also post your times.txt and imu.txtfile.)

jy84-choi commented 2 years ago

When realsense d435i is turned on, imu and camera are not completely synchronized, but I thought that they bring data of a somewhat similar time. If you look at the time.txt and imu.txt files below, you can see that they have data at the same time although there is a slight difference.

======= times.txt ========== 1650957618546289920 1650957618.546289920 1650957618579647744 1650957618.579647744 1650957618613006080 1650957618.613006080 1650957618646362624 1650957618.646362624 1650957618679720192 1650957618.679720192 1650957618713077760 1650957618.713077760 1650957618746436352 1650957618.746436352 1650957618779792640 1650957618.779792640 1650957618813150208 1650957618.813150208 1650957618846507776 1650957618.846507776 1650957618879865344 1650957618.879865344 1650957618913222912 1650957618.913222912 1650957618946580224 1650957618.946580224 1650957618979937792 1650957618.979937792 1650957619013294848 1650957619.013294848 1650957619046651904 1650957619.046651904 1650957619080008960 1650957619.080008960 1650957619113366272 1650957619.113366272 1650957619146723328 1650957619.146723328 1650957619180080384 1650957619.180080384 ... ...

======= imu.txt ========== 1650957618591295744 -0.001701 -0.003494 -0.001733 0.015765 -7.701474 6.218314 1650957618596292352 0.000044 -0.008730 0.001758 0.055764 -7.662160 6.268433 1650957618601289984 -0.001701 -0.000003 0.000013 0.055764 -7.662160 6.268433 1650957618606286592 0.000044 -0.000003 -0.003478 0.055764 -7.662160 6.268433 1650957618611283200 -0.003447 -0.000003 0.000013 0.035688 -7.661873 6.288426 1650957618616279808 -0.001701 -0.000003 0.000013 0.035688 -7.661873 6.288426 1650957618621277696 0.000044 0.003487 0.000013 0.035688 -7.661873 6.288426 1650957618626274048 -0.001701 -0.000003 -0.003478 0.035688 -7.661873 6.288426 1650957618631270656 -0.001701 -0.003494 0.003503 0.055689 -7.662184 6.288163 1650957618636267264 0.000044 -0.000003 0.000013 0.055689 -7.662184 6.288163 1650957618641264896 -0.001701 -0.001749 0.000013 0.055689 -7.662184 6.288163 1650957618646261760 -0.003447 -0.003494 0.000013 0.055781 -7.642186 6.288824 1650957618651258368 -0.001701 -0.000003 -0.003478 0.055781 -7.642186 6.288824 1650957618656254976 -0.005192 0.015705 0.001758 0.055781 -7.642186 6.288824 1650957618661252352 0.000044 0.003487 -0.001733 0.035596 -7.681871 6.287766 1650957618666248960 0.000044 0.001742 0.000013 0.035596 -7.681871 6.287766 1650957618671245824 0.000044 -0.000003 0.000013 0.035596 -7.681871 6.287766 1650957618676242432 -0.003447 -0.000003 0.000013 0.055689 -7.662184 6.288163 1650957618681240064 0.000044 -0.000003 0.001758 0.055689 -7.662184 6.288163 1650957618686236672 0.000044 -0.005239 -0.003478 0.055689 -7.662184 6.288163 ... ...

lukasvst commented 2 years ago

There are two "levels" of IMU-camera synchronization:

The first one is that IMU and camera timestamps are recorded with the same device or otherwise made consistent. This is a prerequisite for running DM-VIO, and I think for the D435i this is the case.
The second level is that the IMU is triggered manually and always records an IMU sample exactly during the timestamp (middle of exposure) of each image. E.g. the VI-Sensor used in the EuRoC dataset does this, but most other visual-inertial sensors don't. For these sensors you can still run DM-VIO, but first you need to add a "fake IMU measurement" for each camera timestamp, by interpolating the neighboring IMU samples. As mentioned, this script does this for you.

So long story short: You need to run python3 interpolate_imu_file --input imu.txt --times times.txt --output pass_this_imu_file_to_dmvio.txt and then it should work.

Please let me know if this solves the issue for you.

jy84-choi commented 2 years ago

There are two "levels" of IMU-camera synchronization:

The first one is that IMU and camera timestamps are recorded with the same device or otherwise made consistent. This is a prerequisite for running DM-VIO, and I think for the D435i this is the case.

The second level is that the IMU is triggered manually and always records an IMU sample exactly during the timestamp (middle of exposure) of each image. E.g. the VI-Sensor used in the EuRoC dataset does this, but most other visual-inertial sensors don't. For these sensors you can still run DM-VIO, but first you need to add a "fake IMU measurement" for each camera timestamp, by interpolating the neighboring IMU samples. As mentioned, this script does this for you.

So long story short: You need to run python3 interpolate_imu_file --input imu.txt --times times.txt --output pass_this_imu_file_to_dmvio.txt and then it should work.

Please let me know if this solves the issue for you.

It works!!!!!!!!! Thank you so much for being patient with me. I think it will be of great help to others if it is included in the readme. It was confirmed that it works well even in environments where pure DSO did not work properly.

lukasvst commented 2 years ago

I'm glad that it helped. Yes, I will include it in the README with a later update.

lukasvst / dm-vio

run dm-vio with realsense d435i #14

original image resolution