search

pals-ttic / sjc

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (CVPR 2023)
https://pals.ttic.edu/p/score-jacobian-chaining
Other
499 stars 15 forks source link

How to get obj files #5

Open Pavankunchala opened 1 year ago

Pavankunchala commented 1 year ago

Hey it is a wonderful repo

I wonder how I can get the output as a 3d model in the obj or fbx format

elloza commented 1 year ago

Same here! It would be a great addition for this demo! Congratulations for the amazing work by the way!

dkebude commented 1 year ago

Hi!

I have been trying to extract an OBJ file with marching cubes from your Voxel NeRF implementation.

My initial idea was to make your code work similar to the Stable-DreamFusion's mesh extraction implemented here: https://github.com/ashawkey/stable-dreamfusion/blob/53d4cef2055234e242d0b9b247c6f41bee7bbf46/nerf/renderer.py#L148L328

I like the utilization of xatlas here since the generated UV becomes very easily editable.

However, I got confused at one point: Here at line 170, https://github.com/ashawkey/stable-dreamfusion/blob/53d4cef2055234e242d0b9b247c6f41bee7bbf46/nerf/renderer.py#L170, the density function retrieves the albedo information as well as the sigmas. The same density function is utilized later at 244.

Furthermore, as far as I understand, they retrieve the mask information from the nvdiffrast after rasterization.

I have read your implementation, and I saw that you generate your own alpha masks and feats/colors.

I would like to utilize your implementation to export mesh in a similar manner and then create a PR for you to review and utilize. However, I was unsure at the points below, could you correct me if I am wrong?

  1. I believe I should utilize the sigmas generated by compute_density_feats function within Vox_RF class.
  2. Albedo channel/colors/feats (I am unsure about the difference here as well) should be retrieved from compute_app_feats function within Vox_RF class.
  3. Mask should be retrieved by running make_alpha_mask function and then utilizing Vox_RF.alphaMask
  4. Is there another way of retrieving sigmas, feats and masks in a single dictionary? I can write a function to retrieve them all from a single function as well. Does state_dict do that?
  5. I am unsure about the binary dilation and erosions within the export_mesh function of Stable-DreamFusion implementation and the nearest neighbors search for anti-aliasing part. Would you like me to implement that part as well?
caseybasichis commented 1 year ago

Did you make any progress exporting an obj from NeRF?

w-hc commented 1 year ago

@caseybasichis thanks for ur interest we are doing a few other things but it's on our mind.

AmanKishore commented 1 year ago

Looks like Sparsefusion just released some of their code! Could be interesting to integrate!

Pavankunchala commented 1 year ago

Any update on this