search

minar09 / cp-vton-plus

Official implementation for "CP-VTON+: Clothing Shape and Texture Preserving Image-Based Virtual Try-On", CVPRW 2020
https://minar09.github.io/cpvtonplus/
MIT License
343 stars 120 forks source link

Result Image's Cloth Edge is not Smooth and Background is Removed #71

Closed rafidgotit closed 3 years ago

rafidgotit commented 3 years ago

I implemented and tested your project just as your instructions with the provided pretrained models. This is Amazing! Now I tried to test it with custom image. Result is ok but not good enough actually.

clothing-problem

The result has basically 3 problems :

  1. Skin Color did not match with my face. I hope that training with my own dataset will solve this issue. That shouldn't be a problem then.
  2. The background is erased. Is there any way not to remove the background? Can you please give me any suggetiones with this?
  3. The main problem is the cloth placement. As you can see in the above picture, cloth on one of my shoulder is not placed well, edge is also not smooth. I guess this is happening for the poor Image-Parsing. I am using Graphonomy with CIHP model to parse image. Can you please suggest me how can I improve this. If the problem is related to image-parsing, can you please give me reference on how to parse image correctly.

Thank you in advance!

minar09 commented 3 years ago

Hi @rafid08 , thank you for your awesome efforts. You have actually summarized the core limitations very nicely, which are also kind of active research topics.

  1. You are right, this is probably the dataset bias.
  2. Yes, you can keep the background very easily, which can be done is various ways. The simplest one might be adding the background after this line (https://github.com/minar09/cp-vton-plus/blob/master/test.py#L172). To do so, you can pass something like im_bg from the model output (similar to https://github.com/minar09/cp-vton-plus/blob/master/cp_dataset.py#L146).
  3. This is actually not the parsing problem, the problem is from the approach itself. Graphonomy is already kinda SOTA for human parsing. The actual issue here is the GMM output which is not fully deformed up to the full target clothing area, and the TOM cannot in-paint the missing parts very well. To solve it, you can try to solve the GMM/TOM issues or maybe try other latest SOTA virtual try-on approaches. Good luck!
rafidgotit commented 3 years ago

Are you using this repository for parsing? https://github.com/Engineering-Course/CIHP_PGN

minar09 commented 3 years ago

Hi @rafid08 , sorry for my late response. We did not use CIHP segmentation at the time of publication. However, I think it's better to use CIHP-PGN for your custom image segmentation, as it has updated labels and better prediction.