Problem for porting VINS-Mobile to Android Platform

[vonzy]

Hello, I am trying porting VINS to Android recently, and I have finished porting VINS-Mobile to Linux cmake version, which was tested successfully with EuRoc_V1_01_easy and EuRoc_MH_01_easy dateset. Several days ago, I collected imu and camera data on Smartisan M1L android mobile phone and have a test with the program, I found the result will get diverged immediately and the cost of nonlinear optimization is about 400-500. So, I wonder what factors may cause this problem, and ask you for help. In the presentation of professor Shen, he mentioned that sensor synchronization and camera-IMU rotation are very important. In Android mobile phone, the timestamp of images may be inaccurate because the image is obtained from a callback function, no timestamp can be obtained when the callback was called, and the timestamp can only be obtained in the function. Also, camera-IMU rotation is [0, -1, 0; -1, 0, 0; 0, 0, -1] in theory, but whether it can be used is not sure. Another question is the configuration of imu parameters, I don't know how to set those parameters. The imu of Smartisan M1L is BMI160. But it seems the parameters are not very important because when I use the parameters of iphone for EuRoc dataset, it also works. All in all, I don't know what to do to solve the problem now, could you please give me some advice? Thank you very much!!!

[PeiliangLi]

1. How did you calibrate your camera and which resolution image have you used?
2. Please log the imu raw data here.

[vonzy]

1. I have calibrated the camera with checkerboard and matlab toolbox, and the resolution is 640x480. the calibration result is

   Focal Length:          fc = [ 570.55125   571.44376 ] +/- [ 18.45453   18.81954 ]
   Principal point:       cc = [ 294.36555   219.09984 ] +/- [ 15.38305   13.54507 ]
   Skew:             alpha_c = [ 0.00000 ] +/- [ 0.00000  ]   => angle of pixel axes = 90.00000 +/- 0.00000 degrees
   Distortion:            kc = [ 0.11629   -0.08567   -0.01099   -0.01631  0.00000 ] +/- [ 0.04120   0.07725   0.00874   0.00979  0.00000 ]
   Pixel error:          err = [ 0.82083   0.77598 ]

2. Here is part of imu raw data:

   223498264914780,0.2734527587890625,0.43896484375,-0.1546630859375,10.327178955078125,2.08624267578125,3.0623626708984375
   223498269980698,0.269195556640625,0.4187164306640625,-0.1280364990234375,10.333160400390625,2.1832122802734375,3.174896240234375
   223498275016099,0.2702484130859375,0.3974151611328125,-0.1013946533203125,10.378646850585938,2.26580810546875,3.3161468505859375
   223498280082017,0.2830352783203125,0.3761138916015625,-0.0715789794921875,10.389419555664063,2.2933502197265625,3.3484649658203125
   223498285117417,0.3011474609375,0.348419189453125,-0.0438690185546875,10.358306884765625,2.3196868896484375,3.3580474853515625
   223498290183335,0.3181915283203125,0.3143310546875,-0.0225677490234375,10.291259765625,2.363983154296875,3.391571044921875
   223498295218735,0.335235595703125,0.2834320068359375,0.00299072265625,10.184722900390625,2.40228271484375,3.3329010009765625
   223498300284653,0.34588623046875,0.2621307373046875,0.027496337890625,10.072189331054688,2.444183349609375,3.223968505859375
   223498305320054,0.34375,0.2621307373046875,0.04986572265625,10.00994873046875,2.465728759765625,3.1246185302734375
   223498310385972,0.3363037109375,0.2855682373046875,0.0690460205078125,10.031494140625,2.4729156494140625,3.05877685546875
   223498315421372,0.313934326171875,0.3388214111328125,0.081817626953125,10.039871215820313,2.4549560546875,3.038421630859375
   223498320487290,0.2819671630859375,0.4091339111328125,0.08502197265625,10.0997314453125,2.47650146484375,3.0096893310546875
   223498325553208,0.2510833740234375,0.4783782958984375,0.0796966552734375,10.129653930664063,2.487274169921875,3.081512451171875
   223498330619126,0.2191162109375,0.5465545654296875,0.07330322265625,10.069793701171875,2.54473876953125,3.1988372802734375
   223498335654526,0.187164306640625,0.5987548828125,0.0669097900390625,9.989593505859375,2.5998077392578125,3.34967041015625
   223498340720444,0.1605377197265625,0.6232452392578125,0.0647735595703125,9.914169311523438,2.6381072998046875,3.4382476806640625
   223498345755845,0.140289306640625,0.6157989501953125,0.0669097900390625,9.800445556640625,2.6536712646484375,3.5687408447265625
   223498350821763,0.1232452392578125,0.5785064697265625,0.075439453125,9.683135986328125,2.64410400390625,3.7387237548828125
   223498355887681,0.116851806640625,0.515655517578125,0.0828857421875,9.525115966796875,2.5543212890625,3.8380889892578125
   223498360923081,0.11578369140625,0.43682861328125,0.09674072265625,9.405410766601563,2.4609375,3.8440704345703125
   223498365988999,0.121124267578125,0.35693359375,0.109527587890625,9.309646606445313,2.2251129150390625,3.8644256591796875
   223498371024400,0.1371002197265625,0.280242919921875,0.1244354248046875,9.187530517578125,1.964141845703125,3.835693359375
   223498376090317,0.163726806640625,0.2067413330078125,0.1382904052734375,9.082183837890625,1.7295074462890625,3.68963623046875
   223498381125718,0.1946258544921875,0.142822265625,0.1500091552734375,8.986419677734375,1.5128326416015625,3.545989990234375
   223498386191636,0.2255096435546875,0.0895538330078125,0.16278076171875,8.974456787109375,1.3967132568359375,3.413116455078125
   223498391227036,0.2606658935546875,0.061859130859375,0.173431396484375,8.970855712890625,1.314117431640625,3.1832733154296875
   223498396292954,0.2968902587890625,0.07037353515625,0.1808929443359375,9.00677490234375,1.2626495361328125,2.9941253662109375
   223498401328355,0.325653076171875,0.11724853515625,0.1883544921875,9.067825317382813,1.2063751220703125,2.78582763671875
   223498406394273,0.3480224609375,0.1939544677734375,0.1904754638671875,9.132461547851563,1.1848297119140625,2.62542724609375
   223498411429673,0.3607940673828125,0.2908935546875,0.18621826171875,9.142044067382813,1.20159912109375,2.5535888671875
   223498416495591,0.368255615234375,0.4102020263671875,0.177703857421875,9.162399291992188,1.2434844970703125,2.4913482666015625
   223498421561509,0.368255615234375,0.5369720458984375,0.173431396484375,9.156417846679688,1.298553466796875,2.5188751220703125
   223498426596909,0.35546875,0.6605377197265625,0.1702423095703125,9.090576171875,1.3847503662109375,2.679290771484375
   223498431662827,0.3416290283203125,0.769195556640625,0.16705322265625,9.017547607421875,1.49847412109375,2.7942047119140625
   223498436698228,0.3160552978515625,0.860809326171875,0.165985107421875,8.924179077148438,1.6277618408203125,2.9725799560546875
   223498441764146,0.2841033935546875,0.92791748046875,0.1723785400390625,8.817626953125,1.6876220703125,3.1976318359375
   223498446799546,0.253204345703125,0.9769134521484375,0.1766357421875,8.665603637695313,1.688812255859375,3.444244384765625
   223498451865464,0.2244415283203125,1.0024871826171875,0.1830291748046875,8.527938842773438,1.636138916015625,3.6393585205078125
   223498456900864,0.1903533935546875,0.99822998046875,0.1936798095703125,8.332809448242188,1.5966339111328125,3.842864990234375
   223498461966782,0.16265869140625,0.9779815673828125,0.2064666748046875,8.120925903320313,1.5104522705078125,4.0332183837890625
   223498467002183,0.1360321044921875,0.9332427978515625,0.220306396484375,7.9485321044921875,1.4422149658203125,4.19122314453125
   223498472068101,0.10833740234375,0.868255615234375,0.2352294921875,7.81207275390625,1.3775634765625,4.2881927490234375
   223498477103501,0.085968017578125,0.790496826171875,0.255462646484375,7.703125,1.298553466796875,4.3612213134765625
   223498482169419,0.0678558349609375,0.7063446044921875,0.2735748291015625,7.666015625,1.2518768310546875,4.4031219482421875
   223498487204819,0.0540008544921875,0.6221923828125,0.2970123291015625,7.654052734375,1.1668701171875,4.442626953125
   223498492270737,0.047607421875,0.53802490234375,0.3172454833984375,7.6480712890625,1.10821533203125,4.41748046875
   223498497336655,0.041229248046875,0.4560089111328125,0.333221435546875,7.6732025146484375,0.992095947265625,4.42706298828125
   223498502372056,0.0401611328125,0.3739776611328125,0.3428192138671875,7.7725677490234375,0.853240966796875,4.3396759033203125
   223498507437974,0.0444183349609375,0.29833984375,0.3534698486328125,7.8527679443359375,0.7562713623046875,4.2834014892578125
   223498512473374,0.0529327392578125,0.221649169921875,0.35986328125,7.9269866943359375,0.648529052734375,4.190032958984375
   223498517539292,0.062530517578125,0.153472900390625,0.3651885986328125,7.9521331787109375,0.56951904296875,4.1086273193359375

[vonzy]

the format of imu data: timestamp,wx,wy,wz,ax,ay,az

[PeiliangLi]

One possible reason is timestamp problem, the callback time may much later than the real image timestamp. The camera calibration result seems not very accurate since the principle point is far away from the image center. Can you tell the imu frame of your device, so I can verify the rotation matrix?

[vonzy]

imu frame? what do you mean?

[wuyuzhang]

@vonzy I'm also trying to move the open-sourced VINS to android platform. How is it going? Have you figured out the drift when rotating problem? Could you please share the no-ros project code on linux to me please and we can share the progress in the future together.

[vonzy]

@wuyuzhang 1.I am trying to improve VO part now, and big rotation problem is still hard for VO/VIO to work out. @FangGet is now working for the porting. 2.Okay, but how can I send the project to you? through e-mail? or IM?

[wuyuzhang]

@vonzy Thanks a lot. My email is: cmuyuzhangwu@gmail.com Let's keep in touch :)

[vonzy]

@wuyuzhang I have send the project to your e-mail address, please check if you receive the e-mail.

[TouqeerAhmad]

@vonzy would you be able to share your VINS-android or VINS-ubuntu with cmake ??

[bileki]

@vonzy @wuyuzhang Could you send me the project by email? I would like to help.