Open xunkar opened 5 months ago
One would also need either a subscription to GPT-4/Gemini-Pro or a local MLLM, which would drastically increase system requirements as models have to be loaded in parallel or decrease time to generate the prompt. What is the benefit you see compared to Fooocus-V2? (modified GPT-2)
The ability to better understand the prompt is pretty powerful in my opinion. Taking a look at the examples provided, it seems capable of understanding things such as where subjects should be placed in relation to each other and in the composition. It also seems to annihiliate attribute bleeding which is a very difficult to deal with as it is.
Recaptionning however is not something I'm attached to, I wonder if this kind of prompt injection so to speak can lead to deviation from the user's request more than help it.
In WildCards, you can add a custom style customization function, and you can generate a variety of styles in batches. namo amitabha.
There is this Omost project of lllyasviel too which kinda works that way, LLM scenery helper, seems on-hold/inactive for now; something similar would be indeed a nice Fooocus-V3 (just noticed there is a thread https://github.com/lllyasviel/Fooocus/discussions/3076).
https://github.com/lllyasviel/Omost
When Fooocus-V2 is a nice enhancer it's always feels like it's throwing random keywords stuff without context or additional scenery.
Is there an existing issue for this?
What would your feature do?
https://github.com/YangLing0818/RPG-DiffusionMaster
Recaptioning, Planning, and Generating with Multimodal LLMs.
Allows finer control of diffusion, with a better understanding of attribute binding and regional placements of subjects, its workflow also allows the user to generate very large images.
Proposed workflow
What is interesting is how invisible it is for the user, so no new UI would be required. However a precise prompt would yield results with higher fidelity.
The only useful thing UI wise would be a checkbox in the advanced option to enable or disable the recaptionning part of the workflow. Then again, the recaptionning is not the most important feature of the project and could be skipped, although useful.
Additional information
No response