bennyguo / instant-nsr-pl

Neural Surface reconstruction based on Instant-NGP. Efficient and customizable boilerplate for your research projects. Train NeuS in 10min!
MIT License
856 stars 84 forks source link

How can I export textured mesh? #34

Open wukailu opened 1 year ago

wukailu commented 1 year ago

The exported .obj file contains only the shape, not the corresponding material and color.

bennyguo commented 1 year ago

Hi, I haven't write code about this. But you could achieve this by (1) query the color (at a fixed viewing direction) for each mesh vertex; (2) rewrite the obj saving code to support color attributes. I'll add this feature when I have time.

adithya1-7 commented 1 year ago

@bennyguo I was not able to figure this part out. Can you tell us which python files have to be modified to achieve the required output? It would be ideal if you provide the code modification snippets with line number. Thanks in advance

iraj465 commented 1 year ago

@bennyguo getting textured mesh would be great! Can you please help on this?

adithya1-7 commented 1 year ago

@bennyguo Found this snippet for colour extraction, but don't know how it applies to current implementation ray_proj = torch.from_numpy(vertices).cuda().float().split(8192 * 32, 0) vertex_col = [model.mesh_color_forward(ray) for ray in ray_proj] vertex_col = (torch.concat(vertex_col).cpu().detach().numpy() * 255.).astype( np.uint8) mesh = trimesh.Trimesh(vertices, triangles, vertex_colors=vertex_col)

iraj465 commented 1 year ago

I was getting some idea from this: https://github.com/xrr-233/Textured-NeuS

But not being able to implement those functions in this repo.

bennyguo commented 1 year ago

Hi guys, I just implemented the textured mesh export, please have a try! If you are resuming from old checkpoints, remember to add export.chunk_size=2097152 export.export_vertex_color=true after your command. For NeRF, you may need a higher marching cubes resolution (~512) to get better vertex color predictions.

Here's a textured chair extracted from NeuS:

image
iraj465 commented 1 year ago

Getting this error. I turned on the flag for export mesh colour. This was working on the last commit. (just the mesh)

Screenshot 2023-04-05 at 10 23 22 AM
bennyguo commented 1 year ago

@iraj465 I also encountered this problem when viewing the mesh using MeshLab, but it seems just a warning. It may be related to using Trimesh to save the mesh. I'll look into it later today.

iraj465 commented 1 year ago

Also, no mesh is coming in obj. Only a bunch of points, whereas it is being trained (in iters images)

Mesh obj:

Screenshot 2023-04-05 at 12 29 39 PM

Iters sample image:

it5000-0

bennyguo commented 1 year ago

Are you training NeRF model? You need to tune model.geometry.isosurface.threshold to get reasonable meshes (in your case set to a smaller value).

iraj465 commented 1 year ago

Yes, i'm training nerf.

This is what i'm getting for the above shoe now.

Screenshot 2023-04-05 at 2 37 11 PM

Earlier i was getting this for the same threshold (in last commit):?

Screenshot 2023-04-05 at 2 40 25 PM
bennyguo commented 1 year ago

Any luck trying a lower threshold?

iraj465 commented 1 year ago

Yeah, lowered the threshold from 5.0 to 3.0. Still the whole shoe is not coming, also the texture seems kinda off.

Screenshot 2023-04-05 at 6 21 02 PM

Image samples. <img width="1067" alt="Screenshot 2023-04-05 at 6 21 02 PM" src="https://user-images.githubusercon

Screenshot 2023-04-05 at 6 26 41 PM Screenshot 2023-04-05 at 6 26 48 PM

tent.com/32464366/230085818-1f9aad21-9fde-41df-a6ee-ce3a3a426d1f.png">

Config used:

    radius: ${model.radius}
    feature_dim: 16
    density_activation: trunc_exp
    density_bias: -1
    isosurface:
      method: mc
      resolution: 1024
      chunk: 2097152
      threshold: 3.0
    xyz_encoding_config:
      otype: HashGrid
      n_levels: 16
      n_features_per_level: 2
      log2_hashmap_size: 19
      base_resolution: 16
      per_level_scale: 1.447269237440378  
    mlp_network_config:
      otype: FullyFusedMLP
      activation: ReLU
      output_activation: none
      n_neurons: 64
      n_hidden_layers: 1
  texture:
    name: volume-radiance
    input_feature_dim: ${model.geometry.feature_dim}
    dir_encoding_config:
      otype: SphericalHarmonics
      degree: 4    
    mlp_network_config:
      otype: FullyFusedMLP
      activation: ReLU
      output_activation: Sigmoid
      n_neurons: 64
      n_hidden_layers: 2

system:
  name: nerf-system
  loss:
    lambda_rgb: 1.
    lambda_distortion: 0.001
  optimizer:
    name: Adam
    args:
      lr: 0.01
      betas: [0.9, 0.99]
      eps: 1.e-15
  scheduler:
    name: MultiStepLR
    interval: step
    args:
      milestones: [10000, 15000, 18000]
      gamma: 0.33

checkpoint:
  save_top_k: -1
  every_n_train_steps: ${trainer.max_steps}

export:
  chunk_size: 2097152
  export_vertex_color: True 

trainer:
  max_steps: 50000
  log_every_n_steps: 200
  num_sanity_val_steps: 0
  val_check_interval: 5000
  limit_train_batches: 1.0
  limit_val_batches: 2
  enable_progress_bar: true 
  precision: 16
bennyguo commented 1 year ago

@iraj465 I think this is related to the new scene center estimation method mentioned in https://github.com/bennyguo/instant-nsr-pl/issues/55#issuecomment-1499944616. In your case it seems the scene center is biased, and your object is not fully wrapped by the bounding box. I'll push a fix later today.

iraj465 commented 1 year ago

I see, the poses are not transformed relatively so the whole scene is shifted. Yeah @bennyguo the fix push would be great. I have also outlined this problem in this issue if you need further details or some testing to do. Issue #60

bennyguo commented 1 year ago

Camera normalization fixed in the latest commit.

bennyguo commented 1 year ago

Camera normalization fixed in the latest commit.