search

google-research / corenet

CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.
Apache License 2.0
104 stars 14 forks source link

Reconstruction of building #4

Open bhowmickarka opened 3 years ago

bhowmickarka commented 3 years ago

Hello, First of all thank you for the amazing work and making it public. I have a small question. I was experimenting with CoreNet for building reconstruction and none of the saved models worked. So, I just wanted to know that whether have you used any of the building data available in ShapeNet for training any of the models? And if not, then if I want to have a model that reconstructs buildings from images, do I need to take care or change any specific part of the code, except for creating a building dataset? Thank you

vitaminSP commented 2 years ago

Hi,

I was experimenting with CoreNet for building reconstruction and none of the saved models worked.

Yes, this is expected. The models learn object priors and the object classes we trained them on don't include buildings. Also, CoReNet is very sensitive to the overall image brightness/gamma, which might further prevent it to work in your case. I noticed this recently and I plan to introduce an optional torchvision.transforms.ColorJitter transform while training to correct for it.

So, I just wanted to know that whether have you used any of the building data available in ShapeNet for training any of the models?

No, unfortunately not. But I am curios to see how CoReNet would perform on such data :)

And if not, then if I want to have a model that reconstructs buildings from images, do I need to take care or change any specific part of the code, except for creating a building dataset?

You should be able to train CoReNet on a new building dataset directly, without modifying the code. Just make sure that the camera matrices in the data are correct, by rendering a few scenes with the data_loading_transformation_and_rendering.ipynb notebook (in the doc folder).

Best regards, Stefan