search

peteanderson80 / Matterport3DSimulator

AI Research Platform for Reinforcement Learning from Real Panoramic Images.
Other
504 stars 130 forks source link

Add depth and object segmentations #13

Open peteanderson80 opened 6 years ago

suhangpro commented 6 years ago

Any plan on when these will be implemented? Thanks for sharing the awesome work!

jotaf98 commented 6 years ago

I'd like to second the request for depth/segmentation!

If I understand correctly, the depth images in the Matterport3D dataset are not aligned with the RGB skybox images. On the other hand, the RGB images that are aligned with the depth images result in panoramas with seams (see discussion in readme here, "Matterport3D support" section). Any idea how to handle this?

peteanderson80 commented 6 years ago

Hi @jotaf98, I enjoyed your MapNet talk, very cool.

Depth / segm is a priority so we will get this done asap. I think we will want to do some color correction and image blending on top of the RGB stitching, to remove the seams. I guess we can just use any near to state of the art codebase for this - I'm not super familiar with this literature so happy to take suggestions on which method to use...

srama2512 commented 5 years ago

Hi @peteanderson80,

Will you be providing an update to incorporate depth map and segmentations soon? If not, can you give me an idea of how easy/hard this is to do and any suggested starting points?

peteanderson80 commented 5 years ago

We have added depth outputs. See the README for details, these are based on the undistorted depth images which are not necessarily perfectly aligned to the skyboxes. I haven't got any immediate plans to do segmentations, but happy to take pull requests :)

srama2512 commented 5 years ago

Thank you for the update! Could you comment on how the imperfect alignment would affect projection of RGB to ground-plane? More specifically, I'm looking to use MapNet on M3D. Will it significantly affect the projection of features from image-plane to ground-plane?

peteanderson80 commented 5 years ago

My expectation is that projecting cnn features to the ground plane would be fine. The location of cnn features in the image is quite coarse anyway (with large receptive fields), and I suspect that depth alignment is mostly off by only a couple of pixels. It might be a bigger issue if the depth maps where being used to generate rgb pointclouds, because the rgb and the xyz might be misaligned a bit.

yellowjs0304 commented 5 years ago

@peteanderson80 Hi, Thank you for sharing nice work. Is there any plan about object segmentation? I really want to do object detection with your simulator.