search

WangYixuan12 / d3fields

[CoRL 24] D^3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation
https://robopil.github.io/d3fields/
MIT License
108 stars 6 forks source link

Info about visualization tool #2

Closed AlbertoRemus closed 11 months ago

AlbertoRemus commented 11 months ago

Hi! really nice work, I particularly like the idea of driving manipulation with AI generated images

Screenshot from 2023-10-13 12-08-46

As depicted in the DinoV2 repo https://github.com/facebookresearch/dinov2/issues/23#issuecomment-1540475277 when we have a bunch of images of the same class to visualize similar parts with the same color it's needed to put them in the same "batch" and perform pca on the flattenized tensor (N x f), with f feature size

So to visualize the Goal image with the same color pattern of the point cloud you needed to perform PCA of the goal together with the other images or did u manage to achieve this directly (and how?)

Thanks in advance for your time!

WangYixuan12 commented 11 months ago

Thank you for your interests in our work! For the PCA, we first fit the feature mapping for one type of object (e.g. shoe). We then use this precomputed PCA to transform high-dim features to lower dimension.

AlbertoRemus commented 11 months ago

@WangYixuan12 thanks for your reply! And how many instances did you use to compute the pca for one class?

WangYixuan12 commented 11 months ago

I just use 4 images. I could upload PCA precomputation code later today.

AlbertoRemus commented 11 months ago

Ah that's great thank you!