search

yifita / DSS

Differentiable Surface Splatting
442 stars 31 forks source link

Using custom renderer to generate mvr data #14

Closed chx125ck closed 2 years ago

chx125ck commented 2 years ago

I'm trying to use a custom renderer to generate the reference data instead of create_mvr_data_from_mesh script. I have some questions.

  1. It seems like light matrices are never used in the trainer, it just creates a light object of the specified type with the default constructor. Are the light data in the data_dict applied elsewhere or are they not being used?
  2. I stored the data_dict (with only one camera as example) as following: {'cameras_type': 'pytorch3d.renderer.cameras.FoVPerspectiveCameras', 'cameras_params': {'znear': 0.01}, 'lights_type': 'DSS.core.lighting.PointLights', 'camera_mat': [[[-0.9967348575592041, 0.0, -0.0807441994547844, 0.0], [0.07663179934024811, 0.3150690793991089, -0.9459699392318726, -15.753456115722656], [0.025439998134970665, -0.9490687251091003, -0.31404030323028564, 169.45343017578125], [0.0, 0.0, 0.0, 1.0]]]} and I did not include the light data since I couldn't find where they are used. And when I run the trainer with this data dict and corresponding image and mask_images, the trainer prints out "transform" from check_grad in rasterizer, and the pointcloud still looks random after 100, 200, 300, and 400 epochs. I couldn't figure out why.
  3. The trainer crashed with the aforementioned reference data when trying to evaluate_3d in trainer because in get_pointcloud in dataset.py, it looks for data_dict["points"] which is not specified to be in a mvr data_dict but was stored by sampling the mesh in create_mvr_data_from_mesh. Why would the points data from the original mesh be needed for training mvr data?
lxxue commented 2 years ago

For the first two questions: I think the lights are used here: https://github.com/yifita/DSS/blob/ce1151b27556a7e2aac18aea2a30ed97f792bb4d/DSS/training/trainer.py#L268-L271

For the third question, I believe they are used for calculating the chamfer distance between the current pointcloud and the ground truth as shown here: https://github.com/yifita/DSS/blob/ce1151b27556a7e2aac18aea2a30ed97f792bb4d/DSS/training/trainer.py#L158-L160

chx125ck commented 2 years ago

Is there anywhere I should look at for situations described in the second question where "transform" is printed out by check_grad?

Also is the chamfer distance used anywhere in the training?

lxxue commented 2 years ago

For the second question, does it still have NaN/Infty after you put in the correct data? I think you should just check why there are NaNs in the point_clouds_filtered.

As far as I know, the chamfer distance is only used for the evaluation of how accurate our predicted point clouds are, not for training.

chx125ck commented 2 years ago

Thank you for answering