search

Xharlie / pointnerf

Point-NeRF: Point-based Neural Radiance Fields
Other
1.09k stars 126 forks source link

Train 2D feature extraction and point representation #8

Open yjcaimeow opened 2 years ago

yjcaimeow commented 2 years ago

Hi @Xharlie and @zexxu ,

I do not understand about "point feature initial" part. Does the model under "init" folder for all dataset? Or for different dataset, I need to train a VGG 2D feature extracting network firstly and seperately? If need, how to train for different dataset.

Can the model under "init" used by other dataset except for dtu? For example, scannet and kitti.

Best, Yingjie

Xharlie commented 2 years ago

hi, the init checkpoint is trained on dtu and can be used for any dataset. It is mentioned in the paper

yjcaimeow commented 2 years ago

Is it ok for outdoor dataset KITTI？

Xharlie commented 2 years ago

i haven't tried it yet.

yuzewang1998 commented 2 years ago

@yjcaimeow Hi,have you success to train the 2D feature extracting network from scratch on ScanNet or KITTI?

SeaBird-Go commented 1 year ago

@Xharlie , Hi, I'm still confused about the train 2D feature extraction and point generation part.

From my understanding and the step-by-step debugging, I find you only optimize the FeatureNet and PointAggregator in this part since the option manual_depth_view=0. (BTW, I don't understand the exact meaning of manual_depth_view when setting to different values).

And from your descriptions in your paper, you use the rendering loss to optimize the networks, but why you set the weight of ray_masked_coarse_raycolor=0.0, and the weight of ray_depth_masked_coarse_raycolor=1.0. So you only use the camera_expected_depth to optimize these networks?

Could you help me understand this part?

Youngju-Na commented 1 year ago

@Xharlie , Hi, I'm still confused about the train 2D feature extraction and point generation part.

From my understanding and the step-by-step debugging, I find you only optimize the FeatureNet and PointAggregator in this part since the option manual_depth_view=0. (BTW, I don't understand the exact meaning of manual_depth_view when setting to different values).

And from your descriptions in your paper, you use the rendering loss to optimize the networks, but why you set the weight of ray_masked_coarse_raycolor=0.0, and the weight of ray_depth_masked_coarse_raycolor=1.0. So you only use the camera_expected_depth to optimize these networks?

Could you help me understand this part?

@Xharlie Same question here. Could anyone give an explanation? Thanks!