search

livepeer / ai-worker

https://livepeer.ai
MIT License
17 stars 26 forks source link

enable multiple prompts for T2I #125

Closed ad-astra-video closed 4 months ago

ad-astra-video commented 4 months ago

This PR enables requests to use multiple prompts supported by SDXL and SD3 with no API updates needed. Diffusers uses the same prompt in all text_encoders if the prompts are not split. Multiple prompts is supported by SDXL and SD3 models and can benefit from specializing the prompts to the different text_ecnoders if tuned properly.

The prompts are split by including a | at the point where want the prompts split.

For SDXL see diffusers pipeline documentation here:https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl Specifically the prompt_2 input to the __call__: https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_xl#diffusers.StableDiffusionXLPipeline.__call__

For SD3 see diffusers pipeline documentation here: https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_3 Specifically the prompt_2 and prompt_3 inputs to the __call__: https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/stable_diffusion_3#diffusers.StableDiffusion3Pipeline.__call__