search

xuebinqin / U-2-Net

The code for our newly accepted paper in Pattern Recognition 2020: "U^2-Net: Going Deeper with Nested U-Structure for Salient Object Detection."
Apache License 2.0
8.57k stars 1.47k forks source link

Combine multiple U-2-Nets or combine multiple pretrained models #289

Open s-e-privalov opened 2 years ago

s-e-privalov commented 2 years ago

Good day! Thank you very much for the unique development! I have a question - with some images there is a problem when removing the background. Of course, i can add my pictures to your training dataset of 10560 pictures and start training from beginning, but this is very long (7 days with 400 epochs on my computer). Maybe it's possible to combine the model you trained with my pretrained model (on my images)? Or maybe there is a way to make two U-2-net networks work in collaboration? One on default pretrained model and the other on my own model? Thank you for answer!

rdutta1999 commented 2 years ago

1) If your test images are similar to the DUTS dataset, you could use the pretrained weights and fine-tune the model instead of training from scratch. 2) You could use Test Time Augmentation and do a weighted sum or average of the results. See here. 3) You could use PyMatting to fine-tune the masks. See here. 4) Train your own model and do an ensemble. Choosing how to ensemble would require manual testing and exploring the fail cases.

s-e-privalov commented 2 years ago

rdutta1999, Thanks a lot for the detailed answer! I will try!

s-e-privalov commented 2 years ago
  1. If your test images are similar to the DUTS dataset, you could use the pretrained weights and fine-tune the model instead of training from scratch.
  2. You could use Test Time Augmentation and do a weighted sum or average of the results. See here.
  3. You could use PyMatting to fine-tune the masks. See here.
  4. Train your own model and do an ensemble. Choosing how to ensemble would require manual testing and exploring the fail cases.

rdutta1999, can you recommend any example of ensembling u2net models? There are many examples on the net with other networks, simpler ones. And how to make an ensemble with u2net is not clear.

rdutta1999 commented 2 years ago
  1. If your test images are similar to the DUTS dataset, you could use the pretrained weights and fine-tune the model instead of training from scratch.
  2. You could use Test Time Augmentation and do a weighted sum or average of the results. See here.
  3. You could use PyMatting to fine-tune the masks. See here.
  4. Train your own model and do an ensemble. Choosing how to ensemble would require manual testing and exploring the fail cases.

rdutta1999, can you recommend any example of ensembling u2net models? There are many examples on the net with other networks, simpler ones. And how to make an ensemble with u2net is not clear.

Sorry for being late to reply. You can try the ensembling methods that have been suggested for other networks (assuming they are for segmentation or matting tasks - in both cases, the model outputs a value in [0,1]).

1) You could simply average them out across n model outputs. 2) Perform a weighted average across n model outputs. You may find these weights using grid search if needed. 3) A hacky method - If a model (or group of models) seem to perform consistently better in certain cases (or regions), for example, Model A better segments out the tires of a car, while Model B better segments out the hood and roof of the car, you can combine the masks by using the upper half of the Model A mask and lower half of the Model B mask.

You may also take a look at the Ensembling section here.