search

ximinng / SVGDreamer

[CVPR 2024] Official implementation for "SVGDreamer: Text Guided SVG Generation with Diffusion Model" https://arxiv.org/abs/2312.16476
https://ximinng.github.io/SVGDreamer-project/
MIT License
219 stars 17 forks source link

About how to use L_reward? #14

Closed Jamie-Cheung closed 4 months ago

Jamie-Cheung commented 4 months ago

In line 564 of svgdreamer/pipelines /SVGDreamer_pipeline.py, you use: phi_output = pipeline.sample(text_prompt,....) However, L_reward is used to update LORA. as you say in paper: Then LoRA model subsequently updates from these reweighted samples. Why not use pipeline.sample_lora(text_prompt,....) ？ Maybe my understanding is wrong. I am looking forward to your reply.

ximinng commented 4 months ago

I understand your confusion. Good question. pipeline.sample_lora is sampling from SD loaded with lora. At this time, the samples are fitting or have already fitted the vector graphics style, and the diversity of SD samples is lost, so we sample from pipeline.sample.

Jamie-Cheung commented 4 months ago

I understand your confusion. Good question. pipeline.sample_lora is sampling from SD loaded with lora. At this time, the samples are fitting or have already fitted the vector graphics style, and the diversity of SD samples is lost, so we sample from pipeline.sample.

But I'm still confused, is L_reward used to update LORA?

ximinng commented 4 months ago

Lines 585~587 clearly state that the update is only happening on LORA.

Jamie-Cheung commented 4 months ago

Lines 585~587 clearly state that the update is only happening on LORA.

Thank you for your response, so the pipeline.sample(text_prompt,....) is sampling images from SD with lora? Because the L_reward is computed based on the images which are generated by pipeline.sample(text_prompt,....) .

ximinng commented 4 months ago

Lines 585~587 clearly state that the update is only happening on LORA.

Thank you for your response, so the pipeline.sample(text_prompt,....) is sampling images from SD with lora? Because the L_reward is computed based on the images which are generated by pipeline.sample(text_prompt,....) .

pipeline.sample samples from SD WITHOUT LoRA.

Jamie-Cheung commented 4 months ago

Thank you very much for your answer. Then I have a question. Is L_reward calculated based on images generated by the SD(w/o Lora), but the gradient is given to the SD (w/ lora).

ximinng commented 4 months ago

Yes, considering the params of other parts of SD are frozen, only LoRA is updated. Other questions are also welcome to exchange~

Jamie-Cheung commented 4 months ago

Yes, considering the params of other parts of SD are frozen, only LoRA is updated. Other questions are also welcome to exchange~

You have solved a big question for me and there are no more questions for now

ximinng commented 4 months ago

Great. Have a nice day~