search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.51k stars 401 forks source link

[Issue]: Cannot load Flux model / master branch (merge ID e7ec07f · 19 hours ago) #3431

Closed mbac closed 3 days ago

mbac commented 4 days ago

Issue Description

Hi,

I'm still getting an error when trying to load a downloaded Flux model, even in the very latest version. The error is:

12:29:34-956851 INFO     Available models: path="/workspace/models/checkpoints" items=3 time=0.02                                      
12:29:34-965439 INFO     Diffuser model downloaded: model="black-forest-labs/FLUX.1-dev"                                               
12:30:22-796889 INFO     Available models: path="/workspace/models/checkpoints" items=3 time=0.03                                      
12:30:29-038737 INFO     Select: model="diffusion_models/black-forest-labs/FLUX.1-dev [0ef5fff789]"                                    
12:30:29-059334 INFO     Autodetect: model="FLUX" class=FluxPipeline                                                                   
                         file="/workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e
                         127f94c8b54bbcced44" size=0MB                                                                                 
12:30:29-133302 ERROR    Diffusers Failed loading model:                                                                               
                         /workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e127f94
                         c8b54bbcced44 The provided pretrained_model_name_or_path "diffusion_models/black-forest-labs/FLUX.1-dev" is   
                         neither a valid local path nor a valid repo id. Please check the parameter.

Please note that I am using a custom models directory, but the relevant one (/workspace/models/diffusion_models) was empty at the time I downloaded the Flux dev model (original version from Hugging Face).

I think there's some sort of problem with the model file path: the printed path here is file="/workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e127f94c8b54bbcced44"

However, that path is the directory which contains the symlink to the model file, not the file itself… Not sure this is expected…

Version Platform Description

Ubuntu 22.04 on RunPod:

12:40:51-664020 INFO     Starting SD.Next                                                                                              
12:40:51-679874 INFO     Logger: file="/workspace/sdnext/sdnext.log" level=INFO size=162138 mode=append                                
12:40:51-686232 INFO     Python version=3.11.9 platform=Linux bin="/workspace/sdnext/venv/bin/python3" venv="/workspace/sdnext/venv"   
12:40:51-866743 INFO     Version: app=sd.next updated=2024-09-13 hash=e7ec07f9 branch=master                                           
                         url=https://github.com/vladmandic/automatic/tree/master ui=main                                               
12:40:52-714915 INFO     Platform: arch=x86_64 cpu=x86_64 system=Linux release=5.15.0-106-generic python=3.11.9                        
12:40:52-719969 INFO     HF cache folder: /root/.cache/huggingface/hub                                                                 
12:40:52-747454 INFO     nVidia CUDA toolkit detected: nvidia-smi present                                                              
12:40:53-775423 INFO     Verifying requirements                                                                                        
12:40:53-790148 INFO     Verifying packages                                                                                            
12:40:53-836280 INFO     Extensions: disabled=[]                                                                                       
12:40:53-839610 INFO     Extensions: enabled=['stable-diffusion-webui-rembg', 'sdnext-modernui', 'sd-webui-agent-scheduler',           
                         'sd-extension-system-info', 'sd-extension-chainner', 'Lora'] extensions-builtin                               
12:40:53-862647 INFO     Extensions: enabled=[] extensions                                                                             
12:40:53-871536 INFO     Startup: quick launch                                                                                         
12:40:53-875911 INFO     Extensions: disabled=[]                                                                                       
12:40:53-879220 INFO     Extensions: enabled=['stable-diffusion-webui-rembg', 'sdnext-modernui', 'sd-webui-agent-scheduler',           
                         'sd-extension-system-info', 'sd-extension-chainner', 'Lora'] extensions-builtin                               
12:40:53-898443 INFO     Extensions: enabled=[] extensions                                                                             
12:40:53-928177 INFO     Command line args: ['--share', '--listen'] share=True listen=True                                             
12:41:44-248675 INFO     Load packages: {'torch': '2.4.1+cu124', 'diffusers': '0.31.0.dev0', 'gradio': '3.43.2'}                       
12:41:48-917231 INFO     VRAM: Detected=44.35 GB Optimization=none                                                                     
12:41:48-937283 INFO     Engine: backend=Backend.DIFFUSERS compute=cuda device=cuda attention="Scaled-Dot-Product" mode=no_grad        
12:41:49-083273 INFO     Device: device=NVIDIA A40 n=2 arch=sm_90 cap=(8, 6) cuda=12.4 cudnn=90100 driver=535.161.08                   
                         535.161.08                                                                                                    
12:41:54-061934 INFO     Using data path: /workspace/podstarter/config/sdnext                                                          
12:41:54-489852 INFO     Available VAEs: path="/workspace/models/vae" items=3                                                          
12:41:54-533653 INFO     Disabled extensions: ['sdnext-modernui']                                                                      
12:41:54-578702 INFO     Available models: path="/workspace/models/checkpoints" items=3 time=0.03                                      
12:41:56-447127 INFO     LoRA networks: available=13 folders=2                                                                         
12:41:58-919762 INFO     Extension: script='extensions-builtin/sd-webui-agent-scheduler/scripts/task_scheduler.py' Using sqlite file:  
                         extensions-builtin/sd-webui-agent-scheduler/task_scheduler.sqlite3                                            
12:41:59-167678 INFO     UI theme: type=Standard name="black-teal"                                                                     
12:42:04-935139 INFO     Local URL: http://localhost:7860/                                                                             
12:42:04-942899 INFO     Share URL: https://69eb791307e5a3f4d2.gradio.live                                                             
12:42:05-291581 INFO     [AgentScheduler] Task queue is empty                                                                          
12:42:05-293781 INFO     [AgentScheduler] Registering APIs                                                                             
12:42:05-542963 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad
                         fp16=True bf16=None optimization=Scaled-Dot-Product                                                           
12:42:05-551300 INFO     Select: model="diffusion_models/black-forest-labs/FLUX.1-dev [0ef5fff789]"                                    
12:42:05-558417 INFO     Autodetect: model="FLUX" class=FluxPipeline                                                                   
                         file="/workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e
                         127f94c8b54bbcced44" size=0MB                                                                                 
12:42:05-588145 ERROR    Diffusers Failed loading model:                                                                               
                         /workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e127f94
                         c8b54bbcced44 The provided pretrained_model_name_or_path "diffusion_models/black-forest-labs/FLUX.1-dev" is   
                         neither a valid local path nor a valid repo id. Please check the parameter.                                   
12:42:05-594620 INFO     Startup time: 71.63 torch=38.32 gradio=8.04 diffusers=1.65 libraries=11.90 ldm=0.19 samplers=0.42             
                         extensions=3.19 face-restore=1.16 upscalers=0.12 networks=0.11 ui-networks=0.63 ui-txt2img=0.17               
                         ui-img2img=0.43 ui-control=0.22 ui-extras=0.06 ui-models=0.45 ui-settings=0.42 ui-extensions=0.78             
                         ui-defaults=0.09 launch=2.52 api=0.22 app-started=0.27 checkpoint=0.15

Relevant log output

2024-09-14 12:40:51,664 | sd | INFO | launch | Starting SD.Next
2024-09-14 12:40:51,679 | sd | INFO | installer | Logger: file="/workspace/sdnext/sdnext.log" level=INFO size=162138 mode=append
2024-09-14 12:40:51,686 | sd | INFO | installer | Python version=3.11.9 platform=Linux bin="/workspace/sdnext/venv/bin/python3" venv="/workspace/sdnext/venv"
2024-09-14 12:40:51,866 | sd | INFO | installer | Version: app=sd.next updated=2024-09-13 hash=e7ec07f9 branch=master url=https://github.com/vladmandic/automatic/tree/master ui=main
2024-09-14 12:40:52,714 | sd | INFO | launch | Platform: arch=x86_64 cpu=x86_64 system=Linux release=5.15.0-106-generic python=3.11.9
2024-09-14 12:40:52,718 | sd | DEBUG | installer | Setting environment tuning
2024-09-14 12:40:52,719 | sd | INFO | installer | HF cache folder: /root/.cache/huggingface/hub
2024-09-14 12:40:52,722 | sd | DEBUG | installer | Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512"
2024-09-14 12:40:52,744 | sd | DEBUG | installer | Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False
2024-09-14 12:40:52,745 | sd | DEBUG | installer | Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True
2024-09-14 12:40:52,747 | sd | INFO | installer | nVidia CUDA toolkit detected: nvidia-smi present
2024-09-14 12:40:53,775 | sd | INFO | installer | Verifying requirements
2024-09-14 12:40:53,790 | sd | INFO | installer | Verifying packages
2024-09-14 12:40:53,833 | sd | DEBUG | installer | Repository update time: Fri Sep 13 16:51:56 2024
2024-09-14 12:40:53,834 | sd | DEBUG | installer | Previous setup time: Sat Sep 14 12:16:41 2024
2024-09-14 12:40:53,836 | sd | INFO | installer | Extensions: disabled=[]
2024-09-14 12:40:53,839 | sd | INFO | installer | Extensions: enabled=['stable-diffusion-webui-rembg', 'sdnext-modernui', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-extension-chainner', 'Lora'] extensions-builtin
2024-09-14 12:40:53,862 | sd | INFO | installer | Extensions: enabled=[] extensions
2024-09-14 12:40:53,864 | sd | DEBUG | installer | Latest extensions time: Sat Sep 14 11:34:16 2024
2024-09-14 12:40:53,869 | sd | DEBUG | installer | Timestamps: version:1726246316 setup:1726316201 extension:1726313656
2024-09-14 12:40:53,871 | sd | INFO | launch | Startup: quick launch
2024-09-14 12:40:53,873 | sd | DEBUG | paths | Register paths
2024-09-14 12:40:53,875 | sd | INFO | installer | Extensions: disabled=[]
2024-09-14 12:40:53,879 | sd | INFO | installer | Extensions: enabled=['stable-diffusion-webui-rembg', 'sdnext-modernui', 'sd-webui-agent-scheduler', 'sd-extension-system-info', 'sd-extension-chainner', 'Lora'] extensions-builtin
2024-09-14 12:40:53,898 | sd | INFO | installer | Extensions: enabled=[] extensions
2024-09-14 12:40:53,924 | sd | DEBUG | installer | Extension preload: {'extensions-builtin': 0.01, '/workspace/podstarter/config/sdnext/extensions': 0.0}
2024-09-14 12:40:53,927 | sd | DEBUG | launch | Starting module: <module 'webui' from '/workspace/sdnext/webui.py'>
2024-09-14 12:40:53,928 | sd | INFO | launch | Command line args: ['--share', '--listen'] share=True listen=True
2024-09-14 12:40:53,930 | sd | DEBUG | launch | Env flags: ['SD_DATADIR=/workspace/podstarter/config/sdnext', 'SD_MODELDIR=/workspace/models']
2024-09-14 12:41:44,248 | sd | INFO | loader | Load packages: {'torch': '2.4.1+cu124', 'diffusers': '0.31.0.dev0', 'gradio': '3.43.2'}
2024-09-14 12:41:48,917 | sd | INFO | shared | VRAM: Detected=44.35 GB Optimization=none
2024-09-14 12:41:48,931 | sd | DEBUG | shared | Read: file="/workspace/podstarter/config/sdnext/config.json" json=42 bytes=2192 time=0.003
2024-09-14 12:41:48,937 | sd | INFO | shared | Engine: backend=Backend.DIFFUSERS compute=cuda device=cuda attention="Scaled-Dot-Product" mode=no_grad
2024-09-14 12:41:49,083 | sd | INFO | shared | Device: device=NVIDIA A40 n=2 arch=sm_90 cap=(8, 6) cuda=12.4 cudnn=90100 driver=535.161.08
535.161.08
2024-09-14 12:41:49,098 | sd | DEBUG | shared | Read: file="html/reference.json" json=52 bytes=28756 time=0.008
2024-09-14 12:41:52,111 | sd | DEBUG | __init__ | ONNX: version=1.19.2 provider=CUDAExecutionProvider, available=['AzureExecutionProvider', 'CPUExecutionProvider']
2024-09-14 12:41:53,872 | sd | DEBUG | sd_hijack | Importing LDM
2024-09-14 12:41:54,059 | sd | DEBUG | webui | Entering start sequence
2024-09-14 12:41:54,061 | sd | INFO | webui | Using data path: /workspace/podstarter/config/sdnext
2024-09-14 12:41:54,093 | sd | DEBUG | webui | Initializing
2024-09-14 12:41:54,489 | sd | INFO | sd_vae | Available VAEs: path="/workspace/models/vae" items=3
2024-09-14 12:41:54,498 | sd | DEBUG | sd_unet | Available UNets: path="/workspace/models/unet" items=1
2024-09-14 12:41:54,513 | sd | DEBUG | model_t5 | Available T5s: path="/workspace/models/clip" items=3
2024-09-14 12:41:54,533 | sd | INFO | extensions | Disabled extensions: ['sdnext-modernui']
2024-09-14 12:41:54,564 | sd | DEBUG | shared | Read: file="/workspace/podstarter/config/sdnext/metadata.json" json=15 bytes=32079 time=0.005
2024-09-14 12:41:54,575 | sd | DEBUG | modelloader | Scanning diffusers cache: folder=/workspace/models/diffusion_models items=1 time=0.01
2024-09-14 12:41:54,578 | sd | INFO | sd_models | Available models: path="/workspace/models/checkpoints" items=3 time=0.03
2024-09-14 12:41:55,742 | sd | DEBUG | webui | Load extensions
2024-09-14 12:41:56,447 | sd | INFO | networks | LoRA networks: available=13 folders=2
2024-09-14 12:41:58,919 | sd | INFO | script_loading | Extension: script='extensions-builtin/sd-webui-agent-scheduler/scripts/task_scheduler.py' Using sqlite file: extensions-builtin/sd-webui-agent-scheduler/task_scheduler.sqlite3
2024-09-14 12:41:58,933 | sd | DEBUG | webui | Extensions init time: 3.19 __init__.py=0.10 Lora=0.15 sd-extension-chainner=0.56 sd-webui-agent-scheduler=1.87
2024-09-14 12:41:59,015 | sd | DEBUG | shared | Read: file="html/upscalers.json" json=4 bytes=2640 time=0.002
2024-09-14 12:41:59,019 | sd | DEBUG | shared | Read: file="extensions-builtin/sd-extension-chainner/models.json" json=24 bytes=2693 time=0.002
2024-09-14 12:41:59,026 | sd | DEBUG | chainner_model | chaiNNer models: path="/workspace/models/chaiNNer" defined=24 discovered=0 downloaded=0
2024-09-14 12:41:59,054 | sd | DEBUG | modelloader | Load upscalers: total=53 downloaded=0 user=0 time=0.11 ['None', 'Lanczos', 'Nearest', 'ChaiNNer', 'SwinIR', 'SD', 'SCUNet', 'RealESRGAN', 'LDSR', 'ESRGAN', 'AuraSR']
2024-09-14 12:41:59,144 | sd | DEBUG | styles | Load styles: folder="/workspace/podstarter/config/sdnext/models/styles" items=288 time=0.09
2024-09-14 12:41:59,162 | sd | DEBUG | webui | Creating UI
2024-09-14 12:41:59,166 | sd | DEBUG | theme | UI themes available: type=Standard themes=12
2024-09-14 12:41:59,167 | sd | INFO | theme | UI theme: type=Standard name="black-teal"
2024-09-14 12:41:59,212 | sd | DEBUG | ui_javascript | UI theme: css="/workspace/sdnext/javascript/black-teal.css" base="sdnext.css" user="None"
2024-09-14 12:41:59,222 | sd | DEBUG | ui_txt2img | UI initialize: txt2img
2024-09-14 12:41:59,396 | sd | DEBUG | ui_extra_networks | Networks: page='model' items=54 subfolders=3 tab=txt2img folders=['/workspace/models/checkpoints', '/workspace/models/diffusion_models', 'models/Reference'] list=0.06 thumb=0.02 desc=0.00 info=0.01 workers=4 sort=Default
2024-09-14 12:41:59,420 | sd | DEBUG | ui_extra_networks | Networks: page='lora' items=13 subfolders=1 tab=txt2img folders=['/workspace/models/loras', '/workspace/models/LyCORIS'] list=0.03 thumb=0.01 desc=0.01 info=0.03 workers=4 sort=Default
2024-09-14 12:41:59,602 | sd | DEBUG | ui_extra_networks | Networks: page='style' items=288 subfolders=1 tab=txt2img folders=['/workspace/podstarter/config/sdnext/models/styles', 'html'] list=0.08 thumb=0.01 desc=0.00 info=0.00 workers=4 sort=Default
2024-09-14 12:41:59,618 | sd | DEBUG | ui_extra_networks | Networks: page='embedding' items=0 subfolders=0 tab=txt2img folders=['/workspace/models/embeddings'] list=0.00 thumb=0.00 desc=0.00 info=0.00 workers=4 sort=Default
2024-09-14 12:41:59,623 | sd | DEBUG | ui_extra_networks | Networks: page='vae' items=3 subfolders=0 tab=txt2img folders=['/workspace/models/vae'] list=0.01 thumb=0.00 desc=0.00 info=0.00 workers=4 sort=Default
2024-09-14 12:41:59,823 | sd | DEBUG | ui_img2img | UI initialize: img2img
2024-09-14 12:42:00,306 | sd | DEBUG | ui_control_helpers | UI initialize: control models=/workspace/models/controlnet
2024-09-14 12:42:01,172 | sd | DEBUG | shared | Read: file="/workspace/podstarter/config/sdnext/ui-config.json" json=0 bytes=2 time=0.003
2024-09-14 12:42:01,381 | sd | DEBUG | theme | UI themes available: type=Standard themes=12
2024-09-14 12:42:02,314 | sd | DEBUG | ui_extensions | Extension list: processed=390 installed=6 enabled=5 disabled=1 visible=390 hidden=0
2024-09-14 12:42:02,501 | sd | DEBUG | webui | Root paths: ['/workspace/sdnext', '/workspace/podstarter/config/sdnext']
2024-09-14 12:42:04,935 | sd | INFO | webui | Local URL: http://localhost:7860/
2024-09-14 12:42:04,942 | sd | INFO | webui | Share URL: https://69eb791307e5a3f4d2.gradio.live
2024-09-14 12:42:04,946 | sd | DEBUG | webui | Gradio functions: registered=1801
2024-09-14 12:42:04,954 | sd | DEBUG | middleware | FastAPI middleware: ['Middleware', 'Middleware']
2024-09-14 12:42:04,965 | sd | DEBUG | webui | Creating API
2024-09-14 12:42:05,291 | sd | INFO | task_runner | [AgentScheduler] Task queue is empty
2024-09-14 12:42:05,293 | sd | INFO | api | [AgentScheduler] Registering APIs
2024-09-14 12:42:05,443 | sd | DEBUG | webui | Scripts setup: ['IP Adapters:0.045', 'XYZ Grid:0.022', 'Face:0.026', 'AnimateDiff:0.013', 'CogVideoX:0.013', 'HDR:0.007', 'LUT Color grading:0.01', 'Prompt enhance:0.006', 'Prompt matrix:0.006', 'Text-to-Video:0.007', 'X/Y/Z Grid:0.02', 'Image-to-Video:0.01', 'LEdits++:0.006', 'Stable Video Diffusion:0.009']
2024-09-14 12:42:05,444 | sd | DEBUG | sd_models | Model metadata: file="/workspace/podstarter/config/sdnext/metadata.json" no changes
2024-09-14 12:42:05,446 | sd | DEBUG | devices | Torch mode: deterministic=False
2024-09-14 12:42:05,541 | sd | DEBUG | devices | Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False
2024-09-14 12:42:05,542 | sd | INFO | devices | Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=None optimization=Scaled-Dot-Product
2024-09-14 12:42:05,549 | sd | DEBUG | modeldata | Model requested: fn=<lambda>
2024-09-14 12:42:05,551 | sd | INFO | sd_models | Select: model="diffusion_models/black-forest-labs/FLUX.1-dev [0ef5fff789]"
2024-09-14 12:42:05,554 | sd | DEBUG | sd_models | Load model: existing=False target=/workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e127f94c8b54bbcced44 info=None
2024-09-14 12:42:05,557 | sd | DEBUG | sd_models | Diffusers loading: path="/workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e127f94c8b54bbcced44"
2024-09-14 12:42:05,558 | sd | INFO | sd_models | Autodetect: model="FLUX" class=FluxPipeline file="/workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e127f94c8b54bbcced44" size=0MB
2024-09-14 12:42:05,586 | sd | DEBUG | model_flux | Loading FLUX: model="diffusion_models/black-forest-labs/FLUX.1-dev" repo="diffusion_models/black-forest-labs/FLUX.1-dev" unet="None" t5="None" vae="Automatic" quant=none offload=none dtype=torch.float16
2024-09-14 12:42:05,587 | sd | DEBUG | model_flux | Loading FLUX: preloaded=[]
2024-09-14 12:42:05,588 | sd | ERROR | sd_models | Diffusers Failed loading model: /workspace/models/diffusion_models/models--black-forest-labs--FLUX.1-dev/snapshots/0ef5fff789c832c5c7f4e127f94c8b54bbcced44 The provided pretrained_model_name_or_path "diffusion_models/black-forest-labs/FLUX.1-dev" is neither a valid local path nor a valid repo id. Please check the parameter.
2024-09-14 12:42:05,593 | sd | DEBUG | script_callbacks | Script callback init time: system-info.py:app_started=0.09 task_scheduler.py:app_started=0.18
2024-09-14 12:42:05,594 | sd | INFO | webui | Startup time: 71.63 torch=38.32 gradio=8.04 diffusers=1.65 libraries=11.90 ldm=0.19 samplers=0.42 extensions=3.19 face-restore=1.16 upscalers=0.12 networks=0.11 ui-networks=0.63 ui-txt2img=0.17 ui-img2img=0.43 ui-control=0.22 ui-extras=0.06 ui-models=0.45 ui-settings=0.42 ui-extensions=0.78 ui-defaults=0.09 launch=2.52 api=0.22 app-started=0.27 checkpoint=0.15
2024-09-14 12:42:05,618 | sd | DEBUG | shared | Save: file="/workspace/podstarter/config/sdnext/config.json" json=42 bytes=2192 time=0.026

Backend

Diffusers

UI

ModernUI

Branch

Master

Model

Other

Acknowledgements

vladmandic commented 3 days ago

this log is pretty clear, i'll handle the issue.

RayHell commented 3 days ago

Same issue for me.

vladmandic commented 3 days ago

this is now fixed in dev branch, service release to master will be next week.

mbac commented 2 days ago

Hi!

Thank you so much for the quick fix. I think I had to perform an additional step for Flux to work: reinstall ONNX with ExecutionProvider.CUDA.

I'm not entirely sure that this step was necessary on my system, though, because I had to restart the RunPod instance just after doing the ONNX reinstall…

vladmandic commented 2 days ago

any onnx issues are very much unrelated to this. and yes, onnx installer is flaky as hell, i wish it wasn't.