Open Animus777 opened 1 month ago
Hi there, thanks for the suggestion. Pushed some commits that maybe helps with this, but not sure if it will work (Kinda ported the A1111 implementation into model management). Can you try and tell me how it goes?
EDIT: Didn't work, reverting the commits for now. Will need to investigate more how to add this feature.
Is there an existing issue for this?
What would your feature do ?
In both UIs A1111 and Forge in Optimizations settings there is an option Cache FP16 weight for LoRA.
It allows you to preserve LoRA quality while keeping checkpoint models in FP8 (to save VRAM and RAM). It works in A1111 but it doesn't work in Forge (FP8 weight setting above it also doesn't work). I assume that's because, unlike A1111, in Forge FP8 mode is enabled not via ui setting but via command line flag --unet-in-fp8-e4m3fn. But I haven't found any command line flags to keep LoRA in FP16:
By digging into files I found that A1111 in modules folder has a file named sd_models.py which seems to have the relevant code about shared.opts.cache_fp16_weight:
I'm not sure if it's a simple copy paste job though... To test if it's working I like to use acceleration LoRA i.e LCM Just set Sampler to LCM, Steps to 4 and CFG to 1. The results are extremely bad in FP8 but if you enable Cache FP16 weight for LoRA they are OK.