Everyday objects | Looking for suggestions

[sixftninja]

Following are the images I provided. Image: Background: Target Background: I used Facebook's Detectron2 to get the mask. The mask is clearly not good. following are the results that I got. Compose: Foreground: Matte: out:

I'm thinking of retraining the Adobe network on all 450 images instead just the non-transparent ones as mentioned in the paper. I'm also in search of a better segmentation model. One that is trained on everyday objects instead of just humans. Please let me know if you're aware of any.

Do you have any other suggestions that I should look into? Please let me know, thanks!

[senguptaumd]

I think everyday objects need a bit different approach than humans. Here are some of my suggestions.

• Let us try to make the test case as easy as possible. Try to capture images with minimal shadows w.r.t background and make sure to use AE/AF lock. For the cup, the background is the wall and the table. Try to make sure that the backgrounds have a different color from the object, not a white cup in front of a white wall.
• The idea is, for humans we have good priors for how they might look. So we rely on priors + the color difference with the background to matte it out. But note that color difference is not always trustworthy. For objects, segmentation will not be so accurate and for many objects it will not be available at all. So we need to focus more on color difference cues.
• With that in mind, I will try to use the whole of Adobe test dataset and compose it into a background. But the whole data augmentation trick needs to vary. As you want to rely more on color differences apply less perturbation to the background, so the captured background is quite as close to the original background. Since the segmentation mask is gonna be quite inaccurate, apply more erosion-dilation-diffusion. Try to tune these and hyper-parameters to make sure the synthetic data training on Adobe dataset is producing decent enough matting.
• Only then continue finetuning on real data.

I hope this helps. Feel free to discuss your progress and I am happy to assist you.

[sixftninja]

Thanks for your suggestions. I'll keep updating the progress here, as and when I get new results. I'm also trying to train on a dataset of everyday objects that I am manually annotating. Once I'm done with that I will share the dataset.