scraed
commented
8 months ago Hi, thanks a lot for the PR! I will read the code and run some tests before merging it into the main branch. Some preliminary comments:
- "Using CFG after the Percent Step" is a brilliant idea! I also observed the same phenomenon: the first 30-40% of the steps alter the image much more than the others. Theoretically, this happens because the deviation of CFG from the Fokker-Planck equation is the largest within the first 30-40% of the steps.
- "Using Characteristic Guidance every n steps": That is an effective way to reduce computational costs. However, I'm concerned about its theoretical soundness. As CHG essentially modifies CFG towards a solution of the Fokker-Planck equation, using CFG and CHG alternately will result in something in between with no clear interpretation. I will test some plots before drawing any conclusions.
- ControlNet fix: I think it is a bug :rofl: . Let me check the ControlNet code and see what happend.
Update:
-
The PR is compatible with previous infotexts, they still yield the same results. very nice!
-
Done some experiments. "Using CFG after the Percent Step" is very effective at perserving much of the correction. "Using Characteristic Guidance every n steps" seems not good at preserving anatomy correction. My intuition is concentrating all CHG steps within a limited range is more reasonable than spread them evenly by alternating with CFG. Maybe we can try to omit the first few CHG steps too.
-
TODO: catch the ControlNet bug
v0xie
First of all, thank you for your excellent extension. I have found that Characteristic Guidance improves image quality and composition by a huge margin.
My main frustration is how long the CHG iteration process can take, given that convergence is not guaranteed. This PR adds two new controls to the UI in an attempt to improve image generation throughput:
"Use CFG after Percent Step" Enables switching from CHG to CFG after a certain percent of steps has passed.
"Use Characteristic Guidance each n steps" Uses CHG every n steps and runs CFG for every other step.
The motivation behind "Use CFG after Percent Step" is the observation that the first 30-40% steps alter the image much more than the remaining steps. By setting it to a value like 0.4, it keeps the first 40% steps using CHG where it would have the most effect, then using CFG the rest of the steps
Alternatively, "Use Characteristic Guidance each n steps" allows alternating between CHG and CFG every n steps. This effectively reduces the number of times CHG runs by a factor of n while still improving image quality.
On a technical note, this PR replaces the forward method with _call_forward, which has the if condition which runs either CHG or CFG if the condition is not met. The default parameter settings will retain the behavior of the extension as it is now.
Please see some examples here:
New slider controls on the UI:
Image comparison between running CHG every step vs only the first 30% of steps:
Image grid comparing generations using both "Use CFG after Percent Step" and "Use Characteristic Guidance each n steps":
Somewhat unrelated to this PR, but definitely notable, is that enabling the extension effectively fixes the high CFG problem with ControlNet InstantID. This works even if no CHG steps are ever run (by setting "Use CFG after Percent Step" to 0).