Make LERF compatible with DINOv2

kerrj / lerf

Code for LERF: Language Embedded Radiance Fields

https://www.lerf.io/

MIT License

668 stars 65 forks source link

Make LERF compatible with DINOv2 #9

Open chungmin99 opened 1 year ago

chungmin99 commented 1 year ago

Added capability to use DINOv2 models

(Using implementation/torchhub as described by https://github.com/facebookresearch/dinov2)

TODOs:

[ ] Compare/contrast DINO vs. DINOv2 features
[ ] Add DINO feature visualization(?) -- for the compare/contrast. Initial implementation cause the colors to be inconsistent across renders, so it feels confusing/disorienting.
[ ] Add compatibility for ViT-B/L/G DINO versions (currently dino-field embedding out hardcoded to 384)

linukc commented 1 year ago

i suggest to move KMeans and torch.pca_low_rank to the _get_outputs_for_camera_raybundle function (aggregate all get_outputs from one image there and after perform dim reduction) to prevent different color palitre on a one image

anas-zafar commented 1 year ago

@chungmin99 , were you able to fully integrate Dinov2 with LERF?