search

wyysf-98 / CraftsMan

CraftsMan: High-fidelity Mesh Generation with 3D Native Diffusion and Interactive Geometry Refiner
https://craftsman3d.github.io/
425 stars 21 forks source link

Details about camera format? #24

Open rfeinman opened 2 months ago

rfeinman commented 2 months ago

Hi,

Thanks for the great code. I am trying to reproduce your diffusion training experiment, however there is only a single object included in the data sample and I cannot find instructions about how to render/prepare more examples. Can you explain the camera format that you are using in rots.npy? I do not see this file included in the official Objaverse-MIX dataset release.

Thank you!

wyysf-98 commented 2 months ago

Hi,

sorry for the unclear description of the data. As we want to make our model more stable to the unperfect mv diffusion prediction, we need some argumentation for the conditioned 4 views images (not the exact 0, 90, 180, 270). It is more convenient for rotating the object rather than the 4 cameras for argumentation, we choose to randomly rotate the objects and save the matrices into 'rots.npy'

Best, Weiyu

rfeinman commented 2 months ago

@wyysf-98 could you explain how the rotation matrix is computed from camera parameters like elevation & azimuth? I see a function called eul2rot (see link below) but I'm not sure what the input parameter theta refers to. Is this the correct function to use? If so, what is the meaning of parameter theta?

Perhaps theta[0] is elevation, and theta[1] is azimuth?

https://github.com/wyysf-98/CraftsMan/blob/9c836e201a122b1f89fc51b3152de870b2ef6ba6/craftsman/data/objaverse.py#L28-L32

wyysf-98 commented 1 month ago

Hi @rfeinman , the code you mentioned is for debugging, which is not used during training.
We only load the rots.npy of the object and transform the camera position like the code linked here.

I will add some comments about the data later and I think training our diffusion model using the single image is also a good idea. Thus, you do not need the camera parameters.

https://github.com/wyysf-98/CraftsMan/blob/9c836e201a122b1f89fc51b3152de870b2ef6ba6/craftsman/data/objaverse.py#L244