Training with masks

[brianneoberson]

Hello,

I am currently attempting to train your model on my own dataset using masks. I am using the colmap format and estimated the normals with your provided script. I first trained without the _--withmask argument and it worked well but the results are not satisfying. So I tried training with the _--withmask argument but it didn't seem to be using the masks. I realized in this line: https://github.com/turandai/gaussian_surfels/blob/5a6715b9f38176daed3e57f6b83e81419a4228c6/scene/dataset_readers.py#L141 that this was not the correct path to my mask data. My masks have exactly the same name as their corresponding images. So I changed this line to mask = load_mask(f'{images_folder}/../mask/{image_name}.png')[None] But now when training with the correct path to my masks and with (but also without) the _--withmask argument I get the following error:

  File "train.py", line 302, in <module>
    training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
  File "train.py", line 145, in training
    loss.backward()
  File "/home/imc/miniconda3/envs/gaussian_surfels/lib/python3.8/site-packages/torch/_tensor.py", line 525, in backward
    torch.autograd.backward(
  File "/home/imc/miniconda3/envs/gaussian_surfels/lib/python3.8/site-packages/torch/autograd/__init__.py", line 267, in backward
    _engine_run_backward(
  File "/home/imc/miniconda3/envs/gaussian_surfels/lib/python3.8/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 2 - got [0, 0, 3] but expected shape compatible with [0, 16, 3] 

Would you have any suggestions as to why this is happening?

[turandai]

Hi, I am sorry, it was a bug with the --with_mask argument, it should be fixed now. Please use the '--use_mask 0' to try run without mask.

[brianneoberson]

Hi, Ok now I only get the error when training with masks (--use_mask 1). It's is still the same error as above. Is there maybe a problem with my masks? They look fine visually but maybe you use a specific format for them?

[turandai]

This error may caused by rendering 0 points, I think the problem may be this mask_prune function. The mask_prune works for our object-centric testing scenes for initialization, but it might over-prune your points. Maybe you can try to comment out this line to see if it helps.

[brianneoberson]

That worked, thanks!