search

facebookresearch / banmo

BANMo Building Animatable 3D Neural Models from Many Casual Videos
Other
539 stars 58 forks source link

Questions about PoseNet #82

Open DevikalyanDas opened 5 months ago

DevikalyanDas commented 5 months ago

Hello, thanks for your great work,

I want to know where is the PoseNet CNN that you used to train the sheep. The pre-trained weights are available but I am not finding the network that uses these weights. Can you please point out where is the training pipeline of the PoseNet as I need it to train a 3D model in a similar setup?

Thank you.

DevikalyanDas commented 5 months ago

Ok it is the 'nerf_root_rts' in banmo.py in line 332. I wanted to know if you have released the random viewpoint generation and random masks augmentation part?

gengshan-y commented 5 months ago

Hi, the training forward function is here, where render_dp(...) contains the data augmentation.

The posenet training is activated here once you pass warmup_pose_ep > 0 (i think we used 10) and pose_cnn_path as "".

Let me know if there are further questions

DevikalyanDas commented 5 months ago

Hello, thanks for your reply. I have one more question about this setup of posenet, if I use Dino rendered dino feature image instead of CSE rendered image, will this setup work?

gengshan-y commented 5 months ago

Note here the assumption to train the posenet is we have a mesh, where each vertex has a K-dimensional semantic features, as learned in the CSE paper.

If you have a geometry (either volume/3dgs/mesh) with learned 3D dino fields, this setup can be applicable.