search

kohya-ss / sd-webui-additional-networks

GNU Affero General Public License v3.0
1.77k stars 296 forks source link

[Feature Request] Auto-detect LoRA dimensions and precision when merging #236

Open TomLucidor opened 9 months ago

TomLucidor commented 9 months ago

The SVD multi-LoRA merging procedure requires "New Rank" and "New Conv Rank", as well as "Merge Precision" and "Save Precision". Those can be automated through reading the metadata of the source LoRA. Ideally, the ranks would use the maximum of the source LoRAs for maximum information preservation. There can be a secondary option for the median/mode of the source, assuming there are weird outliers. As for the "Merge Precision" and "Save Precision" (for both regular Merge LoRA and SVD versions), maximizing the precision to be "float" and then collapsing back to the majority precision (most are either BF16 or FP16) would work.

Q1: is it true that sometimes people want to "fix" a bad 128-dimensional LoRA with this? Q2: What are the benefits in changing "merge precision" to 16 bits when it reduces accuracy? (maybe lower computation cost?)

rafstahelin commented 7 months ago

I'd love a follow up on this