Automated way to remove dynamic object via mask

[smandava98]

Hi. Thanks for the great work. I have a video of an indoor scene with an object bouncing around it but I want to mask away the object. How is this possible?

I have the segmentation masks for it. I saw #101 that mentioned to append an A channel where the 0 means the pixel should not be rendered but I had some questions about that.

Will it still work if in some frames, parts of my scene are blocked by the object but are again visible in future frames if I mask this way?

My goal is the get a fully fleshed 3D representation of the scene minus the object

[yzslab]

Hi, looks like the alpha channel mask only simply set the color of the masked pixels to (0, 0, 0) (scene/cameras.py#L44). I don't think it is suitable for the situation that you mentioned. I have added a dynamic object mask support to gaussian splatting and here is my modification: https://github.com/yzslab/gaussian-splatting/commit/95e7c9282650cac12eb731aa6867d540caa98781. I am still doing experiments on my mask implementation. May be you could try to use my code to train your image set, and tell me whether it work. The mask filename should be IMAGE_FILENAME.png (e.g. the mask of 00001.jpg is 00001.jpg.png), single channel, and the 0 (black) represents the masked pixel, add --masks parameter to train.py to specific the mask image directory.

[smandava98]

Thanks so much @yzslab I'll test on my end and let you know my results soon

[smandava98]

hi @yzslab I don't see any --masks argument in your train.py. Also how is it being read in in train.py? I am not able to get it to read the masks. I made the changes you made and it is still training on the full pixels

[yzslab]

hi @yzslab I don't see any --masks argument in your train.py. Also how is it being read in in train.py? I am not able to get it to read the masks

Hi, the --masks is defined at here. Have you seen the output that tell you how many mask images it read?:

$ python train.py ... --masks ...
Output folder: ...
...
Reading camera 177/177
Read 177 masks
...

[smandava98]

Ah, I forgot to add the args.masks in scene's init.py (I previously added it to the part where it reads in transforms.json instead of sparse). It is loading fine now. Will kick off the training on a few videos to see if it is filling up the scene minus the object now.

[smandava98]

Tested it on a few videos and I can see it is working well. Thanks @yzslab !

[yzslab]

Tested it on a few videos and I can see it is working well. Thanks @yzslab !

Glad to hear that.

[mirlansmind]

@yzslab have you made the repo private? I would really need that asap

[yzslab]

@yzslab have you made the repo private? I would really need that asap

Hi, here is the new link: https://github.com/yzslab/gaussian-splatting/tree/dynamic_mask

[mirlansmind]

@yzslab thank you so much for the prompt reply!

[smoothumut]

@yzslab thank you so much for detailed replies and great solution. 🙇

[LucretiaLuo]

Hi @yzslab , thanks for the repo, it is being helpful to remove the background. However, when I took a closer look at the model, there are ellipsoids in the masked area, some of which are very large in visualization, harming the overal rendering quality. But aftering correcting the mistake in the origianl gs repo as mentioned here https://github.com/graphdeco-inria/gaussian-splatting/issues/283 , there are fewer ellipsoids in the masked area. I wonder if it has something to do with the Gaussian initialization. And is there any way to constrain the gaussians being initialized in the unwanted region? Thanks!