Processing new scenes from EgoExo

[richardrl]

Hi, great work!

I am wondering if you can explain in more detail how you get the colored scenes from EgoExo so we could try running your code overlayed on these scenes.

The pointclouds seem much higher quality than the default pointclouds in semidense_points.csv.gz, and from what I can tell there is no color associated with each point?

For example, this is the semidense points of sfu_cooking_009_5 from egoexo, which looks much lower quality than your scenes:

I noticed a file _cached_filtered_points.npz - perhaps you can explain how you filtered these pointclouds?

And do you build a Gaussian splat somehow?

[brentyi]

Hi @richardrl!

We unfortunately also don't have colors for points from EgoExo:

• The colors of EgoExo points in our results are computed here: https://github.com/brentyi/egoallo/blob/25a95a945fce5a51381332b05a438e78e2f7c2f7/src/egoallo/vis_helpers.py#L158-L166
• The scenes that we visualize with Gaussians are our own captures, which we process, reconstruct, and export using Nerfstudio: https://docs.nerf.studio/quickstart/custom_dataset.html#aria
  • I haven't tried running this process for EgoExo but probably it would work, just with lower fidelity than our (more intentionally) captured sequences.
• The scenes with the colored point clouds in our paper's teaser figure are from unprojecting the dense depth maps from Aria Digital Twin: https://www.projectaria.com/datasets/adt/.

For filtering the EgoExo points, you can check the logic for generating _cached_filtered_points.npz here: https://github.com/brentyi/egoallo/blob/25a95a945fce5a51381332b05a438e78e2f7c2f7/src/egoallo/data/aria_mps.py#L36-L64