search

brentyi / tilted

Canonical Factors for Hybrid Neural Fields @ ICCV 2023
100 stars 2 forks source link

Visualizing method #5

Open chiehwangs opened 1 year ago

chiehwangs commented 1 year ago

Hi Brent,

Firstly, thanks for sharing this excellent and robust work !

Is it possible to share the method of visualizing "the structure-revealing L2-norm of interpolated features" in figure 8 ? I am trying to draw inspiration from your work.

best regards, chieh

brentyi commented 1 year ago

Hi @chiehwangs,

Thanks for your nice words!

For the code that generates these specific heatmaps, you can search the repo for "transform_feature_norms". What's happening here is: we compute the feature norm corresponding to each transform for each sample along the ray, and then alpha composite these norms (with the same weights we would use for compositing RGB). The final visualization is the feature norms a single transformation, which we pick by argmax-ing the norm itself.

The norms are first computed on a per-transform basis in the rendering pipeline. Note that the output shape here is (# rays, # transforms): https://github.com/brentyi/tilted/blob/df4614a79f3fdcaa08727b65b0bd8194ad676441/tilted/nerf/render.py#L338-L361

These norms are then scaled to [0, 1]. We index into only one transform at a time (# rays, # transforms) => (#rays,)), then apply a colormap to convert to RGB for the final visualization: https://github.com/brentyi/tilted/blob/df4614a79f3fdcaa08727b65b0bd8194ad676441/visualize_nerf.py#L188-L199

Hope that's helpful, and please let me know if anything's unclear!