search

jiupinjia / stylized-neural-painting

Official Pytorch implementation of the preprint paper "Stylized Neural Painting", in CVPR 2021.
https://jiupinjia.github.io/neuralpainter/
Creative Commons Zero v1.0 Universal
1.55k stars 261 forks source link

Advice on combining CLIP with the neural renderer #45

Open thuangb opened 2 years ago

thuangb commented 2 years ago

Hi, this is an very interesting and impressive work in using AI for art. Now, as CLIP has created more and more astounding painting from text, I just want to ask some advice, if possible, on combining CLIP and your work together which can be very interesting. Thanks

jiupinjia commented 2 years ago

Hi @thuangb, thanks for your suggestion. Yes, we can definitely add a CLIP loss to our framework. I believe it should work well although I haven't tried that. This would be interesting and worth trying. I will check this out in the near future and probably add this new feature to our method.

thuangb commented 2 years ago

Hi, I just want to ask that what part I have to modify ? I think I must add an new CLIP loss in loss.py and modify the optimization loop in the demo files but I do not familiar with the neural renderer part in your code so I need some advice. Thanks. FYI, I think you can refer to this repo if you plan to add CLIP feature in the future.

jiupinjia commented 2 years ago

Hi @thuangb, since almost everything in the rendering pipeline is differentiable, all you have to do is to add a CLIP similarity loss to the loss function part. I have been working on this recently. Hopefully, this feature will be available very shortly. I will let you know when it is ready.