Different FPS of Replica office0

[jinmaodaomaye2021]

Hi,

Thanks for your great work. Not sure if the algorithm was tested with large view movements.

Since the Replica office0 video has high FPS, I tested the algorithm at different FPS settings.

The algorithm works fine when the FPS was reduced to 1/2, 1/3, 1/5, 1/10 and 1/20 with modifications of parameters. But the algorithm failed to estimate poses at 1/30.

There is still a large view overlapping between frames by visually inspecting the 1/30 FPS data. However, the results are very bad (wrong poses).

In my experiment, I modified the following parameters at 1/30:

• mapping_iters = 500
• mapping_keyframe_every = 1
• mapping-map_every = 1
• mapping_n_pixels = 0.1
• mapping_first_iters = 500
• tracking_iters = 500

Any idea why the algorithm doesn't work for low FPS videos ? What I observed is that if the pose is incorrect from tracking, it is hard to correct in mapping.

[HengyiWang]

Hi @jinmaodaomaye2021, thanks for your interest in our work. Interesting question.

For each scene, there are 2000 frames in total. What you mean is that Co-SLAM can run with only 100 frames and fail at 65 frames. I assume in Replica, the video is simulated with 10-30Hz. Then, in your setting, it would be a 0.3-1Hz video sequence. There are several reasons that might result in losing track in this case:

1. In the tracking process, we only sample 1024 pixels for optimizing camera poses. In your case, part of the samples will be in unobserved regions which would lead to losing track. To address this, I would recommend filtering out outliers from the samples or ensuring that the sampling is performed only in overlapping regions to maintain tracking robustness.

2. The truncation distance is 10cm. You may need to check whether the depth search range is large enough to hit the truncation region.

3. It appears that you set n_pixels=0.1, and you have 65 keyframes in total. This means you are trying to reconstruct the scene with a pixel count of 6.5 frames. I think this amount of pixels can not have enough overlapping for BA. (BTW, if you do not increase #samples for tracking or #keyframe pixels for BA, simply increasing #iters for tracking and mapping might not work very well. 500 tracking and mapping iterations are too much for 1024 tracking samples and 6.5 frame pixels)

I hope these suggestions help. If you have further questions or need more assistance, feel free to ask:)

[jinmaodaomaye2021]

@HengyiWang Thanks for your reply.

Please check the logs below which explains why I chose 100-500 tracking/mapping iterations for low FPS videos (1/20 FPS, tracking_iters=400, mapping_iters=100, tracking_samples=1024, mapping_samples=2048).

I printed out the losses every 10 tracking/mapping iterations. It seems the tracking stage takes many steps to converge for large camera movements (constant_speed=True, camera pose learning rate = 1e-3). If the tracking/mapping stage takes > 100 iterations, the processing time would be very high (3-8 seconds per frame on my machine if mapping_map_every=1, mapping_keyframe_every=1).

tracking iter: 0 loss: 1.481 tracking iter: 10 loss: 1.470 tracking iter: 20 loss: 1.455 tracking iter: 30 loss: 1.438 tracking iter: 40 loss: 1.431 tracking iter: 50 loss: 1.413 tracking iter: 60 loss: 1.401 tracking iter: 70 loss: 1.372 tracking iter: 80 loss: 1.331 tracking iter: 90 loss: 1.265 tracking iter: 100 loss: 1.178 tracking iter: 110 loss: 1.133 tracking iter: 120 loss: 1.118 tracking iter: 130 loss: 1.110 tracking iter: 140 loss: 1.098 tracking iter: 150 loss: 1.087 tracking iter: 160 loss: 1.070 tracking iter: 170 loss: 1.041 tracking iter: 180 loss: 1.011 tracking iter: 190 loss: 0.952 tracking iter: 200 loss: 0.893 tracking iter: 210 loss: 0.792 tracking iter: 220 loss: 0.671 tracking iter: 230 loss: 0.552 tracking iter: 240 loss: 0.446 tracking iter: 250 loss: 0.373 tracking iter: 260 loss: 0.320 tracking iter: 270 loss: 0.278 tracking iter: 280 loss: 0.246 tracking iter: 290 loss: 0.227 tracking iter: 300 loss: 0.222 tracking iter: 310 loss: 0.221 tracking iter: 320 loss: 0.222 tracking iter: 330 loss: 0.215 tracking iter: 340 loss: 0.204 tracking iter: 350 loss: 0.199 tracking iter: 360 loss: 0.187 tracking iter: 370 loss: 0.183 tracking iter: 380 loss: 0.181 tracking iter: 390 loss: 0.182

Best loss: 0.020, Last loss: 0.020

mapping iter 0 loss: 0.148 mapping iter 1 loss: 0.147 mapping iter 2 loss: 0.147 mapping iter 3 loss: 0.141 mapping iter 4 loss: 0.136 mapping iter 5 loss: 0.141 mapping iter 6 loss: 0.146 mapping iter 7 loss: 0.142 mapping iter 8 loss: 0.138 mapping iter 9 loss: 0.135 mapping iter 10 loss: 0.136 mapping iter 11 loss: 0.137 mapping iter 12 loss: 0.135 mapping iter 13 loss: 0.135 mapping iter 14 loss: 0.132 mapping iter 15 loss: 0.134 mapping iter 16 loss: 0.133 mapping iter 17 loss: 0.134 mapping iter 18 loss: 0.132 mapping iter 19 loss: 0.134 mapping iter 20 loss: 0.131 mapping iter 21 loss: 0.127 mapping iter 22 loss: 0.132 mapping iter 23 loss: 0.128 mapping iter 24 loss: 0.136 mapping iter 25 loss: 0.130 mapping iter 26 loss: 0.133 mapping iter 27 loss: 0.126 mapping iter 28 loss: 0.128 mapping iter 29 loss: 0.131 mapping iter 30 loss: 0.127 mapping iter 31 loss: 0.125 mapping iter 32 loss: 0.129 mapping iter 33 loss: 0.128 mapping iter 34 loss: 0.132 mapping iter 35 loss: 0.129 mapping iter 36 loss: 0.125 mapping iter 37 loss: 0.127 mapping iter 38 loss: 0.127 mapping iter 39 loss: 0.130 mapping iter 40 loss: 0.126 mapping iter 41 loss: 0.128 mapping iter 42 loss: 0.124 mapping iter 43 loss: 0.126 mapping iter 44 loss: 0.122 mapping iter 45 loss: 0.128 mapping iter 46 loss: 0.128 mapping iter 47 loss: 0.118 mapping iter 48 loss: 0.123 mapping iter 49 loss: 0.125 mapping iter 50 loss: 0.129 mapping iter 51 loss: 0.125 mapping iter 52 loss: 0.121 mapping iter 53 loss: 0.126 mapping iter 54 loss: 0.128 mapping iter 55 loss: 0.122 mapping iter 56 loss: 0.125 mapping iter 57 loss: 0.125 mapping iter 58 loss: 0.121 mapping iter 59 loss: 0.122 mapping iter 60 loss: 0.121 mapping iter 61 loss: 0.120 mapping iter 62 loss: 0.125 mapping iter 63 loss: 0.118 mapping iter 64 loss: 0.118 mapping iter 65 loss: 0.117 mapping iter 66 loss: 0.120 mapping iter 67 loss: 0.123 mapping iter 68 loss: 0.120 mapping iter 69 loss: 0.124 mapping iter 70 loss: 0.122 mapping iter 71 loss: 0.117 mapping iter 72 loss: 0.119 mapping iter 73 loss: 0.120 mapping iter 74 loss: 0.120 mapping iter 75 loss: 0.114 mapping iter 76 loss: 0.118 mapping iter 77 loss: 0.120 mapping iter 78 loss: 0.115 mapping iter 79 loss: 0.121 mapping iter 80 loss: 0.117 mapping iter 81 loss: 0.119 mapping iter 82 loss: 0.116 mapping iter 83 loss: 0.122 mapping iter 84 loss: 0.113 mapping iter 85 loss: 0.113 mapping iter 86 loss: 0.120 mapping iter 87 loss: 0.118 mapping iter 88 loss: 0.118 mapping iter 89 loss: 0.117 mapping iter 90 loss: 0.111 mapping iter 91 loss: 0.119 mapping iter 92 loss: 0.112 mapping iter 93 loss: 0.118 mapping iter 94 loss: 0.116 mapping iter 95 loss: 0.116 mapping iter 96 loss: 0.115 mapping iter 97 loss: 0.111 mapping iter 98 loss: 0.112 mapping iter 99 loss: 0.112

Update current pose

add keyframe: 2

[HengyiWang]

@jinmaodaomaye2021, thanks for providing the log. I assumed you have tried different lr for this. Then, can you try 1 to see if that help? A simple way is to select several frames, and manually set a sample region that ensures the overlapping.