search

crockwell / Cap3D

[NeurIPS 2023] Scalable 3D Captioning with Pretrained Models
https://huggingface.co/datasets/tiange/Cap3D
218 stars 13 forks source link

Script for sampling the colored point cloud #8

Closed Tycho-Xue closed 1 year ago

Tycho-Xue commented 1 year ago

@crockwell Hi, thanks for the excellent work! Do you mind sharing your script for extracting the colored point cloud for Objaverse?

tiangeluo commented 1 year ago

hi @Tycho-Xue ,

I just cleaned the code for extracting colorful point clouds. You can refer to extract_pointcloud.py and npz2ply.py in https://github.com/crockwell/Cap3D/tree/main/text-to-3D/shap-e.

It is implemented based on the code of shap-E, and thus, you need to compile shapE codes (plz cite shapE if you use this part of codes !).

Plz feel free to let me know if you need further assistance.

Tycho-Xue commented 1 year ago

@tiangeluo, thanks for the quick response! Let me try it

Tycho-Xue commented 1 year ago

@tiangeluo , Hi, thanks for the pointers! it seems a bit slow, just curious, have you figured out a way to speed this process up?

tiangeluo commented 1 year ago

Hi @Tycho-Xue,

First, I'd like to kindly mention that our dataset provides 660k colorful point clouds for Objaverse shapes: https://huggingface.co/datasets/tiange/Cap3D/tree/main/PointCloud_zips along with the point clouds stored as .pt files, which provide faster loading speed: https://huggingface.co/datasets/tiange/Cap3D/tree/main/PointCloud_pt_zips.

The slow speed is mainly caused by the farthest point sampling (https://github.com/crockwell/Cap3D/blob/37a47fd2f1312f874398c30893c7f5898dc443b2/text-to-3D/shap-e/shap_e/rendering/point_cloud.py#L155-L207). You can install a cuda version FPS to accelerate (https://github.com/crockwell/Cap3D/blob/37a47fd2f1312f874398c30893c7f5898dc443b2/text-to-3D/shap-e/shap_e/rendering/point_cloud.py#L13-L19).

Qi-Zhangyang commented 1 year ago

Impressive Work I want to ask, from Objaverse mesh to point cloud. You first render the mesh to image and then get the point cloud instead of getting the point cloud from mesh directly? Thanks

tiangeluo commented 1 year ago

Hi @Qi-Zhangyang , yes, you understand is correct. More precisely, the program will render objects into multiple rgbd images and use camera rays to gather colorful visible points.

https://github.com/crockwell/Cap3D/blob/493a7e3798bf9c8b2900f27e8fc0636a39d2d760/text-to-3D/shap-e/shap_e/rendering/point_cloud.py#L46-L102

https://github.com/crockwell/Cap3D/blob/493a7e3798bf9c8b2900f27e8fc0636a39d2d760/text-to-3D/shap-e/shap_e/util/data_util.py#L197-L223

https://github.com/crockwell/Cap3D/blob/493a7e3798bf9c8b2900f27e8fc0636a39d2d760/text-to-3D/shap-e/shap_e/util/data_util.py#L113-L127