Question about original plane

facebookresearch / 3D-Vision-and-Touch

When told to understand the shape of a new object, the most instinctual approach is to pick it up and inspect it with your hand and eyes in tandem. Here, touch provides high fidelity localized information while vision provides complementary global context. However, in 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. In this paper, we study this problem and present an effective chart-based approach to fusing vision and touch, which leverages advances in graph convolutional networks. To do so, we introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects. Our results show that (1) leveraging both vision and touch signals consistently improves single-modality baselines, especially when the object is occluded by the hand touching it; (2) our approach outperforms alternative modality fusion methods and strongly benefits from the proposed chart-based structure; (3) reconstruction quality boosts with the number of grasps provided; and (4) the touch information not only enhances the reconstruction at the touch site but also extrapolates to its local neighborhood.

Other

68 stars 14 forks source link

Hi, I have a further question about the model. During the initialization of the model, a plane is defined by:

width = .0218 - 0.00539 y_z = torch.arange(dim).cuda().view(dim, 1).expand(dim, dim).float() y_z = torch.stack((y_z, y_z.permute(1, 0))).permute(1, 2, 0) plane = torch.cat((torch.zeros(dim, dim, 1).cuda(), y_z), dim=-1) self.orig_plane = (plane / float(dim) - .5) * width

Based on my understanding, the prediction of the model is based on the deformation of this plane, may I ask what's the meaning of "width"? It is a parameter related to your simulator? Thanks so much for your help!

facebookresearch / 3D-Vision-and-Touch

Question about original plane #7