search

kerrj / lerf

Code for LERF: Language Embedded Radiance Fields
https://www.lerf.io/
MIT License
668 stars 65 forks source link

"multiphrase" output renderer #12

Closed acube3 closed 1 year ago

acube3 commented 1 year ago

When I try to follow the instructions, I only get relevancy_0 output renderer. However, The controls shown in the demo video are totally different that what I get. Also, I see that the demo video uses multiphrase output renderer. How am I supposed to get the same thing running on my side?

chungmin99 commented 1 year ago

Hi! Thank you for your interest :)

You should be able to visualize the relevancy maps using relevancy_0 option (as in https://github.com/kerrj/lerf#using-lerf). This renderer has the same functionality as the multiphrase output renderer in the demo video.

(This codebase was refactored for smooth integration with the updated nerfstudio viewer -- the demo video was taken before the changes)

acube3 commented 1 year ago

I do see kind of a heat map while using relevancy_0 option. However, it is not same as the demo video where the pixels were highlighted and segmentation is visible on top of RGB renderings. Also, could you please tell me what settings you used to create the demo video (one at the bottom of the page?) and what GPU was used for rendering. Was video made faster for better visualization? Thank you in advance.

kerrj commented 1 year ago

This PR implements the rgb+relevancy composited visualization, which should match the video on the website. The video is not sped up, it's running on a server A100 (the desktop analog is a 3090). On slower GPUs it will probably be more laggy when dragging the viewpoint around. Also consider lowering the viewer "Max Res" if you find it takes too long to render full-resolution frames.

Also see the "relevancy map normalization" section of the README here for some details on visualizing the raw relevancies.