Does it support onlineTsdf fusion？

[llianxu]

Hi！Thanks for your impressive work！ I found that in your code， you first fuse all images into TSDF， and then train the sdf field. I want to know whether it supports online TSDF fusion and sdf field training. For example, add the img one by one into tsdf fusion, and the train process only sample in existed img.

[sunjiahaovo]

I have the same question. Looking forward to an answer!

[theNded]

Hey @llianxu and @sunjiahaovo thanks for the interest!

If you use RGB-D I'm pretty sure it just works. I may need to expose more APIs to actively resize/reparameterize the hash grids for potential expansion, but that is easily achievable. I can make PRs if necessary.

For monocular images, it could be more challenging since we do not have a consistent scale in advance. The best attempt right now could be to first collect several dozen/hundred frames for initialization, and estimate the scale of incoming images from ray-casted depths and input monocular depth. That might introduce additional artifacts though.

[llianxu]

@theNded Thank you very much for your response. I have tried online frame-by-frame fusion of TSDF and training, but found that the results were poor. I simply modified the dataloader method in the data provider and the order of training scene reconstruction, but the performance was not satisfactory. I would like to know if it is due to the reparameterization and resizing that you mentioned. Could you please tell me where to make the modifications and the reasons behind them? Alternatively, could you submit a pull request (PR) for me? I’m currently rushing on a paper with a deadline. Thank you very much!

[theNded]

If there is no error at runtime, then reparameterization should not be necessary. a) If you are using RGB, I think scale is still the predominant problem, and there is no trivial way to resolve it; b) If you are using RGB-D, a large batch size may help. I'm currently using 4096, but it could be as large as 65536 in the previous version. I haven't tried it here though.

[llianxu]

@theNded Thank you very much for your reply！I use RGB-D data. I would like to know if the dense grid is updated incrementally along with TSDF fusion. I am using the fuse_frame function for incremental updates, but I have noticed that as the training progresses, the PSNR gradually decreases. I will try the large batch size first

[theNded]

They are incrementally updated. However, from my knowledge, the result of (incremental) fusion and the result of optimization may be at different local optimals. That may introduce additional problems.

[llianxu]

Thank you very much for your patient responses. I'll go and think about it some more.

[theNded]

I made a PR for larger batch sizes. It works fine with batch_size=65536 and eats <6G memory on a standard dataset. Hopefully it will help (a little bit).

[llianxu]

I will immediately test this. Thank you very much!

[small-zeng]

I have halted the training process, the TSDF fusion continues. However, I have encountered an issue where I am unable to export the mesh dynamically during the TSDF fusion process. My goal is to obtain real-time updates of the mesh changes. The details of the problem are outlined below:

1、when I fuse dataset and export mesh as original code:

final mesh is ：

2、when I fuse dataset and export mesh online:

final mesh is partially missing:

The only difference between these two methods lies in the intermediate step of mesh extraction. Could it be that the marching_cubes function does not support real-time extraction during the TSDF fusion process?