Open wukailu opened 1 year ago
Hi, I haven't write code about this. But you could achieve this by (1) query the color (at a fixed viewing direction) for each mesh vertex; (2) rewrite the obj saving code to support color attributes. I'll add this feature when I have time.
@bennyguo I was not able to figure this part out. Can you tell us which python files have to be modified to achieve the required output? It would be ideal if you provide the code modification snippets with line number. Thanks in advance
@bennyguo getting textured mesh would be great! Can you please help on this?
@bennyguo Found this snippet for colour extraction, but don't know how it applies to current implementation
ray_proj = torch.from_numpy(vertices).cuda().float().split(8192 * 32, 0)
vertex_col = [model.mesh_color_forward(ray) for ray in ray_proj]
vertex_col = (torch.concat(vertex_col).cpu().detach().numpy() * 255.).astype( np.uint8)
mesh = trimesh.Trimesh(vertices, triangles, vertex_colors=vertex_col)
I was getting some idea from this: https://github.com/xrr-233/Textured-NeuS
But not being able to implement those functions in this repo.
Hi guys, I just implemented the textured mesh export, please have a try! If you are resuming from old checkpoints, remember to add export.chunk_size=2097152 export.export_vertex_color=true
after your command. For NeRF, you may need a higher marching cubes resolution (~512) to get better vertex color predictions.
Here's a textured chair extracted from NeuS:
Getting this error. I turned on the flag for export mesh colour. This was working on the last commit. (just the mesh)
@iraj465 I also encountered this problem when viewing the mesh using MeshLab, but it seems just a warning. It may be related to using Trimesh to save the mesh. I'll look into it later today.
Also, no mesh is coming in obj. Only a bunch of points, whereas it is being trained (in iters images)
Mesh obj:
Iters sample image:
Are you training NeRF model? You need to tune model.geometry.isosurface.threshold
to get reasonable meshes (in your case set to a smaller value).
Yes, i'm training nerf.
This is what i'm getting for the above shoe now.
Earlier i was getting this for the same threshold (in last commit):?
Any luck trying a lower threshold?
Yeah, lowered the threshold from 5.0 to 3.0. Still the whole shoe is not coming, also the texture seems kinda off.
Image samples. <img width="1067" alt="Screenshot 2023-04-05 at 6 21 02 PM" src="https://user-images.githubusercon
tent.com/32464366/230085818-1f9aad21-9fde-41df-a6ee-ce3a3a426d1f.png">
Config used:
radius: ${model.radius}
feature_dim: 16
density_activation: trunc_exp
density_bias: -1
isosurface:
method: mc
resolution: 1024
chunk: 2097152
threshold: 3.0
xyz_encoding_config:
otype: HashGrid
n_levels: 16
n_features_per_level: 2
log2_hashmap_size: 19
base_resolution: 16
per_level_scale: 1.447269237440378
mlp_network_config:
otype: FullyFusedMLP
activation: ReLU
output_activation: none
n_neurons: 64
n_hidden_layers: 1
texture:
name: volume-radiance
input_feature_dim: ${model.geometry.feature_dim}
dir_encoding_config:
otype: SphericalHarmonics
degree: 4
mlp_network_config:
otype: FullyFusedMLP
activation: ReLU
output_activation: Sigmoid
n_neurons: 64
n_hidden_layers: 2
system:
name: nerf-system
loss:
lambda_rgb: 1.
lambda_distortion: 0.001
optimizer:
name: Adam
args:
lr: 0.01
betas: [0.9, 0.99]
eps: 1.e-15
scheduler:
name: MultiStepLR
interval: step
args:
milestones: [10000, 15000, 18000]
gamma: 0.33
checkpoint:
save_top_k: -1
every_n_train_steps: ${trainer.max_steps}
export:
chunk_size: 2097152
export_vertex_color: True
trainer:
max_steps: 50000
log_every_n_steps: 200
num_sanity_val_steps: 0
val_check_interval: 5000
limit_train_batches: 1.0
limit_val_batches: 2
enable_progress_bar: true
precision: 16
@iraj465 I think this is related to the new scene center estimation method mentioned in https://github.com/bennyguo/instant-nsr-pl/issues/55#issuecomment-1499944616. In your case it seems the scene center is biased, and your object is not fully wrapped by the bounding box. I'll push a fix later today.
I see, the poses are not transformed relatively so the whole scene is shifted. Yeah @bennyguo the fix push would be great. I have also outlined this problem in this issue if you need further details or some testing to do. Issue #60
Camera normalization fixed in the latest commit.
Camera normalization fixed in the latest commit.
The exported
.obj
file contains only the shape, not the corresponding material and color.