fail using texture.py

[Juanjlp1109]

Describe the bug Hello, first of all thanks to the authors. The problem I have is with the texture, I scanned a corner of the office, the walls and the floor came out with their colors fine, but the small objects came out with reddish tints and the table instead of coming out totally white, came out with a brown layer on top of it.

To Reproduce Steps to reproduce the behavior:

1. First process the data with Colmap in Nerfstudio like this: ns-process-data images --data /home/lidarit/sdfstudio/prueba_monica --output-dir /home/lidarit/sdfstudio/prueba_monica

2. obtained depth and normal maps with omnidata: python scripts/datasets/process_nerfstudio_to_sdfstudio.py --data prueba_monica --output-dir prueba_monica/omnidata --scene-type indoor --data-type colmap --mono-prior --crop-mult 1 --omnidata-path omnidata/omnidata_tools/torch/ --pretrained-models omnidata/omnidata_tools/torch/pretrained_models

3. train a monosdf model with the following parameters: ns-train monosdf --pipeline.model.sdf-field.use-grid-feature True --pipeline.model.sdf-field.hidden-dim 256 --pipeline.model.sdf-field.num-layers 2 --pipeline.model.sdf-field.num-layers-color 2 --pipeline.model.sdf-field.use-appearance-embedding True --pipeline.model.sdf-field.geometric-init True --pipeline.model.sdf-field.inside-outside True --pipeline.model.sdf-field.bias 0.8 --pipeline.model.sdf-field.beta-init 0.1 --pipeline.datamanager.train-num-images-to-sample-from 1 --pipeline.datamanager.train-num-times-to-repeat-images 0 --trainer.steps-per-eval-image 5000 --pipeline.model.background-model none --vis wandb --experiment-name colmap-andres --pipeline.model.mono-depth-loss-mult 0.1 --pipeline.model.mono-normal-loss-mult 0.05 --pipeline.datamanager.train-num-rays-per-batch 2048 --machine.num-gpus 1 sdfstudio-data --data prueba_monica/omnidata --include_mono_prior True --skip_every_for_val_split 30

4. after the training I extracted the mesh with the following command ns-extract-mesh --load-config outputs/colmap-andres/monosdf/2023-07-14_174149/config.yml --output-path outputs/colmap-andres/monosdf/2023-07-12_133453/mesh_monica.ply --bounding-box-min -0.5 -0.5 -0.5 --bounding-box-max 0.5 0.5 0.5 --resolution 1024 --marching_cube_threshold 0.001 --create_visibility_mask True

5. Extract the texture with the script texture.py 1008 python scripts/texture.py --load-config outputs/colmap-andres/monosdf/2023-07-14_174149/config.yml --input-mesh-filename outputs/colmap-andres/monosdf/2023-07-12_133453/mesh_monica.ply --output-dir outputs/colmap-andres/monosdf/2023-07-14_174149

Expected behavior a texture with the correct colors in its entirety

Screenshots

Additional context here is the drive with the images that I used: https://drive.google.com/drive/folders/1vQIjXfHVD18MV7MPvwHSpTn-U8jVIg6U?usp=sharing

[niujinshuchong]

@Juanjlp1109 I think the currently texturing using the inverse surface normal as the viewing direction to query the appearance network (see here https://github.com/autonomousvision/sdfstudio/blob/master/nerfstudio/exporter/texture_utils.py#L311-L315). However, these view directions are not the same as view directions in the training images so it's doing some extrapolation instead of interpolation and therefore the resulting color is not the same as training colors.

[Juanjlp1109]

@niujinshuchong I undestand, is there a way to solve it?

[niujinshuchong]

@Juanjlp1109 I think you need to replace the viewing direction with one of the directions of training images.

[Juanjlp1109]

I suppose the image I choose must be one that represents the whole scene? it must have some specific characteristic?

[niujinshuchong]

@Juanjlp1109 You are not restricted to use only one image. Maybe the easiest way to find the closest training view directions based on its cosine similarity to the inverse surface normal.

[Juanjlp1109]

Can you explain more detail please, I don't understand totally

[niujinshuchong]

@Juanjlp1109 You can try something like this:

forward_direction = torch.tensor([0, 0, 1]).reshape(1, 3, 1) # principle direction of each camera
training_directions = c2ws[:, :3,:3] @ forward_direction # [N, 3], principle direction of each camera in world coordinate

viewing_directions = -surface_normal # [M, 3]
cosine_distance = torch.sum(viewing_directions[:, None, :], training_directions[None, :, :], dim=-1) # [M, N]
index = cosine_distance.argmax(dim=-1) # [M] find the closest direction
viewing_directions = training_directions[index] # [M, 3]

[lbrianza]

@Juanjlp1109 You can try something like this:

forward_direction = torch.tensor([0, 0, 1]).reshape(1, 3, 1) # principle direction of each camera
training_directions = c2ws[:, :3,:3] @ forward_direction # [N, 3], principle direction of each camera in world coordinate

viewing_directions = -surface_normal # [M, 3]
cosine_distance = torch.sum(viewing_directions[:, None, :], training_directions[None, :, :], dim=-1) # [M, N]
index = cosine_distance.argmax(dim=-1) # [M] find the closest direction
viewing_directions = training_directions[index] # [M, 3]

hi @niujinshuchong ! thank you for this great project, it's really awesome.

I'm having the same problem as described by @Juanjlp1109 - the mesh looks great, but the texture looks confusing when I train with monosdf (while it looks perfectly fine when I train using neus-facto). I am a bit confused about your piece of code - I don't see c2ws anywhere inside texture_utils.py, how can I extract it? Also, what is 'surface_normal'? (I don't see it in the code neither). Thank you!

[niujinshuchong]

@lbrianza c2ws are the camera to world transformation and should be read from the data manager. surface_normal is the normal at the vertices.

@Juanjlp1109 Did you solve the problem? Would be great if you can share your solution?

[440981]

@niujinshuchong Hello, I also do not understand the specific need to modify which code?

[440981]

The model exported using python scripts/texture.py cannot be compared with the model in the viewer at all. How can I solve it?