search

YangLing0818 / RPG-DiffusionMaster

[ICML 2024] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (RPG)
https://proceedings.mlr.press/v235/yang24ai.html
MIT License
1.7k stars 99 forks source link

support diffusers #36

Open akk-123 opened 9 months ago

akk-123 commented 9 months ago

The current code is a bit bloated, can you provide more concise diffusers code?

YangLing0818 commented 9 months ago

Thanks for your attention! In the near future, we will provide diffusers-based code.

YangLing0818 commented 7 months ago

The current code is a bit bloated, can you provide more concise diffusers code?

Thanks for your waiting, we now have supported diffusers.

akk-123 commented 7 months ago

@YangLing0818 I test it, but I get a black img, I found latents here is all NaN

I test with model runwayml/stable-diffusion-v1-5 and use gpt-4

ZhaochenYu0201 commented 7 months ago

Thank you for your feedback. If you get black image, it may because of the NSFW filter. The prompt you have given or generated image might be closer to one of the not allowed content. In such cases you can get black image as output . You can use different prompt and try different random seed to test if it is becasue of the NSFW filter. And it could also be becasue of your device , which may not support half-precision, try to use full-precision instead.

akk-123 commented 7 months ago

Not because it’s NSFW, I set safe_checker is None, and I sad that latents is also NAN, I try fp32, It's same

ZhaochenYu0201 commented 7 months ago

Thank you for your feedback ! I have tested the code in RPG.py on different devices including A100-80G, A100-40G and RTX4090. For me, model runwayml/stable-diffusion-v1-5 in full-precision and half-precision works well with A100, and RTX4090 works well with full-precision. Would you mind running diffusers-cli env and pasting the output from the command? Also, please let us know the GPU and the code you are using . Thank you! You can also refer to the issue which has been handled here huggingface/diffusers#2153