search

DLR-RM / BlenderProc

A procedural Blender pipeline for photorealistic training image generation
GNU General Public License v3.0
2.74k stars 443 forks source link

ScanNet or SUNRGBD datasets with BlenderProc #987

Open wakasu opened 11 months ago

wakasu commented 11 months ago

Describe your feature request

I have used for creating datasets for COCO instance segmentation and Linemod. I really appreciate the work that the developers have done with this tool. I am working on a monocular 3D object detection project and I need to use either ScanNet or SUNRGBD datasets. However, it seems like these datasets are not supported by BlenderProc.

I would like to know if anyone has worked on this problem before and if they could share their approach or code with me. If not, I might start working on it myself if I have some spare time.

Describe a possible solution

No response

cornerfarmer commented 11 months ago

Hey @wakasu,

how exactly do you want to use these datasets? Both provide rgb-d sequences which are not directly suitable for blenderproc. Or do you want to load the provided 3D reconstructions? If yes, then this should be possible by using the bproc.loader.load_obj(path) method, where path points to e.g. the ply files from ScanNet.

wakasu commented 11 months ago

Hey @cornerfarmer

Currently, I would like to have the capability to export the dataset in SUNRGBD or ScanNet format. As for input data, I'm flexible and can use either my own .blend scenes or, if this project progresses further, I can also work on 3D reconstruction using indoor point cloud datasets such as S3DIS.

cornerfarmer commented 11 months ago

Ah okay, it was not clear to me that you want to export data in that formats. That should be easy to do. The blenderproc renderer returns a list of numpy arrays for each image, so you can export them in any format you want. SUNRGBD only seems to be a specific folder structure containing the explicit images. So here you just need to export the numpy arrays as images into the correct path. For ScanNet you need to write the images in a custom .sens format. Thats a bit more difficult, as you need to put them into a struct complementary to how the .sens readings works (see https://github.com/ScanNet/ScanNet/blob/master/SensReader/python/SensorData.py)

wakasu commented 11 months ago

Wow, thank you for the message and for providing the link to the code for reference. I will begin working on this shortly.