drprojects
commented
1 year ago Hi @JonasSchult,
Thanks for your message ! I remember meeting you at CVPR'22 we talked about your to-be-published ECCV paper Mask3D (congrats by the way) :wink:
So, about Dynamic-Size Image-Batching, you will find it as a composition of several dataset transforms. If you look at conf/data/segmentation/multimodal/s3disfused-sparse.yaml for instance:
CenterRoll: rotates the image sphere around the vertical axis to place the mapped pixels in the center of the image (for S3DIS equirectangular images only). This prepares the next stepsPickImagesFromMappingAreaandCropImageGroups. You should discard this transform for datasets where images are not 360° (eg KITTI-360, ScanNet)PickImagesFromMappingArea: drops images which do not contain enough mapped pixels. This is optional but avoids encoding images which are non-informative for the 3D points at hand.CropImageGroups: crops each image with a bounding box including all its mapped pixels. This produces images of various sizes, depending on how far the image is located from the 3D points. But since we prefer processing images in same-size batches, the bounding boxes are adjusted to the nearest box among a family of boxes (eg[64, 64], [128, 64], [128, 128], [256, 128], [256, 256], [512, 256], [512, 512], ...). The output of this transform is anImageDataand not aSameSettingImageData. AnImageDatais a holder class to carry a list ofSameSettingImageDatawith different sizes, precisely whatCropImageGroupsoutputs.PickImagesFromMemoryCredit: selects a subset of the images based on their size and how informative their mappings are. This transform ensures that the number of images in the batch does not bypass a chosen "pixel credit". This is an augmentation and caps the GPU memory usage.
As you seem to have well-understood, for datasets such as ScanNet, CropImageGroups does not make sense because we take the whole room at once and the camera is located inside the room. However, if you have a look at conf/data/segmentation/multimodal/scannet-sparse.yaml, you will see we still use PickImagesFromMemoryCredit to mitigate GPU memory use. By default, PickImagesFromMemoryCredit will assign a "pixel cost" to each image, based on its size. But you can also specify k_coverage to balance-out the image cost based on how much additional coverage (ie how many unseen points) each image contains.
Hope that helps !
JonasSchult
Hi Damien!
Thanks for releasing the code of your great paper! :)
Your Dynamic-Size Image-Batching of Section 3.3 sounds very interesting to me!
However, I could not find the code for it. Could you point me to the right file?
In particular, I'm very interested in the part where you iteratively select images with the probability proportional to the number of pixels vs. number of newly seen points in the cloud, up until a certain budget.
Do you still use the cropping technique for ScanNet even though you operate on full rooms and do not do any sampling of spheres as in S3DIS? In that case, do you still use Dynamic-Size Image-Batching for ScanNet?
Best, Jonas