Chuan-10
commented
2 months ago Hi, sorry for the late reply and thank you for your attention to our work.
There are mainly two methods to calculate relevancy. The first method is splatting the semantic features onto 2D images, similar to rendering RGB, which is adopted by LERF and most other works. In our work, we calculate the relevancy using this method, and specific code can be found in render_mask.py. Because it uses rasterization, it is really fast. The second method, as you mentioned, involves calculating it for every Gaussian. However, this approach can be time-consuming due to the large number of Gaussians. While it may yield results, the semantic features associated with each Gaussian may be less accurate compared to splatted ones, due to alpha blending. Regarding efficiency, the bottleneck primarily arises from decoding semantic features into codebook indices. To strike a balance between speed and accuracy, I suggest experimenting with a smaller decoder MLP and shorter semantic features on Gaussians. Alternatively, limiting the number of Gaussians during training is feasible, and this functionality is already implemented in our code.
Actually, the visualization scripts are about how to draw the relevancy heatmaps according to the relevancy rendered by render_mask.py and are not related to the calculation method itself. Anyway, if you need it, please let me know, and I will clean it up and post it here.
Hello!
Thanks for your nice work. I saw in your TODO list the task of "Add relevancy maps of text queries visualization scripts." I wonder if this means the ability to show this directly on 3DGS (same as Fig.1 in your paper). If so, could you please give hints how to do so effectively and share some tips. Because now I'm think of checking every single Gaussian and extract its semantic feature and convert it as CLIP to calculate the relevancy, it seems for me that this would take a long time.
Thanks in advance