[Issue]: v-prediction not working on safetensors models?

bmhRi3kXAFSw commented 3 months ago

Issue Description

After an update a few months ago I have no longer been able to use v-prediction models successfully. I am using diffusers, IPEX and linux.

The current recommendation for use of vpred sd 1.5 models on diffusers backend is to convert them to the diffusers model format. however:

this is a time consuming process, especially since most models are not distributed in this format; this conversion is not available from within the UI and instead I need to set up a python venv with diffusers (huge) or run some scripts to piggyback off the existing one
i cannot seem to get the webui to even detect things in diffusers format (what folder do they go in?)

additionally, manually using the built-in settings to override scheduler, etc. does not seem to work either: setting "override model prediction type" to vpred and checking "rescale betas with zero terminal snr" still produces results characteristic of running a vpred model incorrectly

previously i was able to get these models working by hacking in yaml config file loading for diffusers (see this patch file I wrote a while ago, in another issue #2914 sdmodelsfix.patch.txt).

however, this no longer appears to be working; still get bad vpred errors.

I know this is a weird/unusual configuration but I am at a loss for how to continue to get this working. I do not know that much about the internals of SD or vladmatic, or how model loading really works, and would appreciate some help (especially just enabling config loading for diffusers backend/flat files, like it is for the normal backend)

there may be some other issues that are caused by a compounding of other problems elsewhere but I did have this configuration working a few months ago and at some point it broke. I have also tested normal config models and they work fine

I also tested this on a fresh clean install of course. it happens with both dev and master branch

Version Platform Description

Arch Linux Python 3.11.9 ARC A770 IPEX Backend 2.1.20 Kernel Linux-zen 6.9.7

Relevant log output

Activate python venv: /home/bmhRi3kXAFSw/automatic/venv
Launch: venv/bin/python3
INFO     Starting SD.Next                                                                                               
INFO     Logger: file="/home/bmhRi3kXAFSw/automatic-new/sdnext.log" level=DEBUG size=64 mode=create                 
INFO     Python version=3.11.9 platform=Linux bin="/home/bmhRi3kXAFSw/automatic-new/venv/bin/python3"               
         venv="/home/bmhRi3kXAFSw/automatic-new/venv"                                                               
INFO     Version: app=sd.next updated=2024-07-07 hash=dddf0f2a branch=dev                                               
         url=https://github.com/vladmandic/automatic.git/tree/dev ui=dev                                                
INFO     Latest published version: 9e7bb45ca9eb1333c125ebf4dfcaf812cac1ae10 2024-07-05T01:10:32Z                        
INFO     Platform: arch=x86_64 cpu= system=Linux release=6.9.7-zen1-1-zen python=3.11.9                                 
DEBUG    Setting environment tuning                                                                                     
INFO     HF cache folder: /home/bmhRi3kXAFSw/.cache/huggingface/hub                                                      
DEBUG    Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512"                                     
DEBUG    Torch overrides: cuda=False rocm=False ipex=True diml=False openvino=False                                     
DEBUG    Torch allowed: cuda=False rocm=False ipex=True diml=False openvino=False                                       
INFO     Python version=3.11.9 platform=Linux bin="/home/bmhRi3kXAFSw/automatic-new/venv/bin/python3"               
         venv="/home/bmhRi3kXAFSw/automatic-new/venv"                                                               
INFO     Intel OneAPI Toolkit detected                                                                                  
INFO     Verifying requirements                                                                                         
INFO     Verifying packages                                                                                             
DEBUG    Repository update time: Sun Jul  7 11:44:36 2024                                                               
INFO     Startup: standard                                                                                              
INFO     Verifying submodules                                                                                           
DEBUG    Submodule: extensions-builtin/sd-extension-chainner / main                                                     
DEBUG    Submodule: extensions-builtin/sd-extension-system-info / main                                                  
DEBUG    Submodule: extensions-builtin/sd-webui-agent-scheduler / main                                                  
DEBUG    Submodule: extensions-builtin/sdnext-modernui / dev                                                            
DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-rembg / master                                            
DEBUG    Submodule: modules/k-diffusion / master                                                                        
DEBUG    Submodule: wiki / master                                                                                       
DEBUG    Register paths                                                                                                 
DEBUG    Installed packages: 223                                                                                        
DEBUG    Extensions all: ['sdnext-modernui', 'sd-webui-agent-scheduler', 'sd-extension-system-info',                    
         'stable-diffusion-webui-rembg', 'sd-extension-chainner', 'Lora']                                               
DEBUG    Running extension installer:                                                                                   
         /home/bmhRi3kXAFSw/automatic-new/extensions-builtin/sd-webui-agent-scheduler/install.py                    
DEBUG    Running extension installer:                                                                                   
         /home/bmhRi3kXAFSw/automatic-new/extensions-builtin/stable-diffusion-webui-rembg/install.py                
DEBUG    Extensions all: []                                                                                             
INFO     Extensions enabled: ['sdnext-modernui', 'sd-webui-agent-scheduler', 'sd-extension-system-info',                
         'stable-diffusion-webui-rembg', 'sd-extension-chainner', 'Lora']                                               
INFO     Verifying requirements                                                                                         
DEBUG    Setup complete without errors: 1720373716

Backend

Diffusers

Branch

Master

Model

SD 1.5

Acknowledgements

[X] I have read the above and searched for existing issues
[X] I confirm that this is classified correctly and its not an extension issue

vladmandic commented 3 months ago

additionally, manually using the built-in settings to override scheduler, etc. does not seem to work either: setting "override model prediction type" to vpred and checking "rescale betas with zero terminal snr" still produces results characteristic of running a vpred model incorrectly

i'll check this - its been a long time since i've touched any v-pred model.

previously i was able to get these models working by hacking in yaml config file loading for diffusers

diffusers stopped using yaml config files, now everything is in json format (still in /configs folder), so if you managed to hack yaml before, you might be able to do the same to json files.

btw, for testing, can you post a link to one such model? i can find it as well, but this makes sure we're using the same thing.

bmhRi3kXAFSw commented 3 months ago

most of the fluffyrock models and their descendents use vpred https://civitai.com/models/92450 thank you for your help!!!!

vladmandic commented 3 months ago

i've loaded model from your link, set settings -> sampler -> prediction type: v-prediction and it just works. model is veeery sensitive - i somewhat decent outputs at 768x1024 resolution, cfg scale 7.0 and rescale guidance 1.0 at lower resolution or cfg scale or rescale, it's just blobs. alternatively instead of high rescale guidance, you can set attention guidance which triggers pag pipeline.

bmhRi3kXAFSw commented 3 months ago

I do not think this model is that sensitive. maybe it is now, but it was not always that way. If I select generations that I did before months ago and copy their parameters identically, they go from correct output to blobs in recent versions. I think there's some kind of a regression, maybe from (other) config parameters that are not loading correctly. For example the params block in config has the following:

params:
    parameterization: "v"
    linear_start: 0.00085
    linear_end: 0.0120
    num_timesteps_cond: 1
    log_every_t: 200
    timesteps: 1000
    first_stage_key: "jpg"
    cond_stage_key: "txt"
    image_size: 64
    channels: 4
    cond_stage_trainable: false   # Note: different from the one we trained before
    conditioning_key: crossattn
    monitor: val/loss_simple_ema
    scale_factor: 0.18215
    use_ema: False

I can override the vpred setting, but what about the others? Are these only used in training and not inference? There are other config settings in the yaml files as well. My thought is that some parameter is not loading and causing the instability.

Without the config patch, it looks like leaving the v-pred override to default and turning off "full quality" fixes the model, even at low resolutions. I will experiment with config settings and update this post with what works, but I think it's full quality setting.

vladmandic commented 3 months ago

if turning full quality off produces good results, that's quite possibly related to scale-factor param in config as that is vae config item.

bmhRi3kXAFSw commented 3 months ago

Are configs being loaded on the master branch with diffusers on? If not, is there an scale factor override somewhere for me to test? (or is full quality unimportant enough that I can just leave it off and not use it?)

It also appears that after a restart, I need to set sampler to Euler A to get things to work right. After doing so, I can set it to Default (??)

Edit: Full quality does work at low resolutions?! I've played around with a lot of settings and I can't seem to reproduce the blobbiness - it happens on startup, and then after changing a few options around and getting a "successful" generation, I can't seem to get it to happen again.

I am having an issue with img2img at high resolutions producing poor results, but I think that's model-related

vladmandic commented 2 days ago

this thread went quiet few months ago - is this still a valid issue?

vladmandic / automatic