[Issue]: Flux models don't want to load with CLIPTextModel error

Issue Description

When trying to load a Flux model an error is returned as follows:
SingleFileComponentError: Failed to load CLIPTextModel. Weights for this component appear to be missing in the checkpoint. Please load the component before passing it in as an argument to from_single_file.
text_encoder = CLIPTextModel.from_pretrained('...') pipe = StableDiffusionPipeline.from_single_file(, text_encoder=text_encoder)
Tried changing diffusers to 0.30.0 but got same error. This was produced on the latest repo version, newly generated VENV, RTX 4090.
Version Platform Description

Using VENV: A:\sd.next\venv 14:07:33-368044 INFO Starting SD.Next 14:07:33-371543 INFO Logger: file="A:\sd.next\sdnext.log" level=DEBUG size=65 mode=create 14:07:33-373044 INFO Python version=3.10.11 platform=Windows bin="A:\sd.next\venv\Scripts\python.exe" venv="A:\sd.next\venv" 14:07:33-520567 INFO Version: app=sd.next updated=2024-08-31 hash=bab17a0b branch=master url=https://github.com/vladmandic/automatic/tree/master ui=main 14:07:33-938117 INFO Platform: arch=AMD64 cpu=AMD64 Family 25 Model 33 Stepping 2, AuthenticAMD system=Windows release=Windows-10-10.0.22631-SP0 python=3.10.11 14:07:33-944117 INFO HF cache folder: C:\Users\roy_p.cache\huggingface\hub 14:07:33-970117 INFO nVidia CUDA toolkit detected: nvidia-smi present 14:07:34-105646 INFO Verifying requirements 14:07:41-142551 INFO Verifying packages 14:07:41-178051 DEBUG Repository update time: Sat Aug 31 22:57:34 2024 14:07:41-179050 INFO Startup: standard 14:07:41-180050 INFO Verifying submodules 14:07:44-373500 INFO Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'sdnext-modernui', 'stable-diffusion-webui-rembg']
Relevant log output

Using VENV: A:\sd.next\venv
14:07:33-368044 INFO     Starting SD.Next
14:07:33-371543 INFO     Logger: file="A:\sd.next\sdnext.log" level=DEBUG size=65 mode=create
14:07:33-373044 INFO     Python version=3.10.11 platform=Windows bin="A:\sd.next\venv\Scripts\python.exe"
                         venv="A:\sd.next\venv"
14:07:33-520567 INFO     Version: app=sd.next updated=2024-08-31 hash=bab17a0b branch=master
                         url=https://github.com/vladmandic/automatic/tree/master ui=main
14:07:33-938117 INFO     Platform: arch=AMD64 cpu=AMD64 Family 25 Model 33 Stepping 2, AuthenticAMD system=Windows
                         release=Windows-10-10.0.22631-SP0 python=3.10.11
14:07:33-939616 DEBUG    Setting environment tuning
14:07:33-944117 INFO     HF cache folder: C:\Users\roy_p\.cache\huggingface\hub
14:07:33-945117 DEBUG    Torch allocator: "garbage_collection_threshold:0.80,max_split_size_mb:512"
14:07:33-956616 DEBUG    Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False
14:07:33-957618 DEBUG    Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True
14:07:33-970117 INFO     nVidia CUDA toolkit detected: nvidia-smi present
14:07:34-105646 INFO     Verifying requirements
14:07:34-108147 WARNING  Package version mismatch: diffusers 0.30.0 required 0.30.2
14:07:34-109648 INFO     Install: package="diffusers==0.30.2" mode=pip
14:07:34-110648 DEBUG    Running: pip="install --upgrade diffusers==0.30.2"
14:07:41-142551 INFO     Verifying packages
14:07:41-178051 DEBUG    Repository update time: Sat Aug 31 22:57:34 2024
14:07:41-179050 INFO     Startup: standard
14:07:41-180050 INFO     Verifying submodules
14:07:42-978811 DEBUG    Submodule: extensions-builtin/sd-extension-chainner / main
14:07:43-035325 DEBUG    Submodule: extensions-builtin/sd-extension-system-info / main
14:07:43-087824 DEBUG    Submodule: extensions-builtin/sd-webui-agent-scheduler / main
14:07:43-170838 DEBUG    Git detached head detected: folder="extensions-builtin/sdnext-modernui" reattach=main
14:07:43-172337 DEBUG    Submodule: extensions-builtin/sdnext-modernui / main
14:07:43-240855 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-rembg / master
14:07:43-296358 DEBUG    Submodule: modules/k-diffusion / master
14:07:43-374866 DEBUG    Git detached head detected: folder="wiki" reattach=master
14:07:43-375866 DEBUG    Submodule: wiki / master
14:07:43-407879 DEBUG    Register paths
14:07:43-474379 DEBUG    Installed packages: 175
14:07:43-475878 DEBUG    Extensions all: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sdnext-modernui', 'stable-diffusion-webui-rembg']
14:07:43-667404 DEBUG    Running extension installer: A:\sd.next\extensions-builtin\sd-webui-agent-scheduler\install.py
14:07:44-056956 DEBUG    Running extension installer:
                         A:\sd.next\extensions-builtin\stable-diffusion-webui-rembg\install.py
14:07:44-372499 DEBUG    Extensions all: []
14:07:44-373500 INFO     Extensions enabled: ['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sdnext-modernui', 'stable-diffusion-webui-rembg']
14:07:44-375002 INFO     Verifying requirements
14:07:44-376000 DEBUG    Setup complete without errors: 1725192464
14:07:44-379499 DEBUG    Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
14:07:44-380501 DEBUG    Starting module: <module 'webui' from 'A:\\sd.next\\webui.py'>
14:07:44-382000 INFO     Command line args: ['--models-dir', 'A:\\stable-diffusion-webui\\models', '--autolaunch',
                         '--debug'] models_dir=A:\stable-diffusion-webui\models autolaunch=True debug=True
14:07:44-383500 DEBUG    Env flags: []
14:07:49-768770 INFO     Load packages: {'torch': '2.4.0+cu124', 'diffusers': '0.30.2', 'gradio': '3.43.2'}
14:07:50-889913 INFO     VRAM: Detected=23.99 GB Optimization=none
14:07:50-894412 DEBUG    Read: file="config.json" json=35 bytes=1569 time=0.000
14:07:50-895913 INFO     Engine: backend=Backend.DIFFUSERS compute=cuda device=cuda attention="Scaled-Dot-Product"
                         mode=no_grad
14:07:50-938423 INFO     Device: device=NVIDIA GeForce RTX 4090 n=1 arch=sm_90 cap=(8, 9) cuda=12.4 cudnn=90100
                         driver=560.94
14:07:50-939923 DEBUG    Read: file="html\reference.json" json=52 bytes=29118 time=0.000
14:07:51-229461 DEBUG    ONNX: version=1.19.0 provider=CUDAExecutionProvider, available=['AzureExecutionProvider',
                         'CPUExecutionProvider']
14:07:51-434985 DEBUG    Importing LDM
14:07:51-450985 DEBUG    Entering start sequence
14:07:51-451984 INFO     Using models path: A:\stable-diffusion-webui\models
14:07:51-454484 DEBUG    Initializing
14:07:51-484984 INFO     Available VAEs: path="A:\stable-diffusion-webui\models\VAE" items=2
14:07:51-486485 DEBUG    Available UNets: path="A:\stable-diffusion-webui\models\UNET" items=0
14:07:51-487984 INFO     Disabled extensions: []
14:07:51-489988 DEBUG    Read: file="cache.json" json=2 bytes=1088 time=0.000
14:07:51-491989 DEBUG    Read: file="metadata.json" json=59 bytes=153879 time=0.001
14:07:51-495988 DEBUG    Scanning diffusers cache: folder=A:\stable-diffusion-webui\models\Diffusers items=0 time=0.00
14:07:51-497489 INFO     Available models: path="A:\stable-diffusion-webui\models\Stable-diffusion" items=27 time=0.01
14:07:51-729521 DEBUG    Load extensions
14:07:51-790025 INFO     Extension: script='extensions-builtin\Lora\scripts\lora_script.py'
                         [2;36m14:07:51-787021[0m[2;36m [0m[34mINFO    [0m LoRA networks: [33mavailable[0m=[1;36m32[0m
                         [33mfolders[0m=[1;36m2[0m
14:07:52-329091 INFO     Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using
                         sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
14:07:52-344091 DEBUG    Extensions init time: 0.61 sd-webui-agent-scheduler=0.50
14:07:52-357590 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2672 time=0.000
14:07:52-359091 DEBUG    Read: file="extensions-builtin\sd-extension-chainner\models.json" json=24 bytes=2719 time=0.000
14:07:52-361091 DEBUG    chaiNNer models: path="A:\stable-diffusion-webui\models\chaiNNer" defined=24 discovered=0
                         downloaded=0
14:07:52-363090 DEBUG    Upscaler type=ESRGAN folder="A:\stable-diffusion-webui\models\ESRGAN" model="4x-UltraSharp"
                         path="A:\stable-diffusion-webui\models\ESRGAN\4x-UltraSharp.pt"
14:07:52-364591 DEBUG    Upscaler type=ESRGAN folder="A:\stable-diffusion-webui\models\ESRGAN" model="4x_NMKD-Siax_200k"
                         path="A:\stable-diffusion-webui\models\ESRGAN\4x_NMKD-Siax_200k.pth"
14:07:52-365590 DEBUG    Upscaler type=ESRGAN folder="A:\stable-diffusion-webui\models\ESRGAN" model="ESRGAN_4x"
                         path="A:\stable-diffusion-webui\models\ESRGAN\ESRGAN_4x.pth"
14:07:52-369090 DEBUG    Load upscalers: total=56 downloaded=6 user=3 time=0.02 ['None', 'Lanczos', 'Nearest',
                         'ChaiNNer', 'AuraSR', 'ESRGAN', 'LDSR', 'RealESRGAN', 'SCUNet', 'SD', 'SwinIR']
14:07:52-386090 DEBUG    Load styles: folder="A:\stable-diffusion-webui\models\styles" items=288 time=0.02
14:07:52-389590 DEBUG    Creating UI
14:07:52-391095 DEBUG    UI themes available: type=Modern themes=32
14:07:52-392094 INFO     UI theme: type=Modern name="IlluZn-Superuser"
14:07:52-399594 DEBUG    UI theme: css="extensions-builtin\sdnext-modernui\themes\IlluZn-Superuser.css" base="base.css"
                         user="None"
14:07:52-402594 DEBUG    UI initialize: txt2img
14:07:52-482602 DEBUG    Networks: page='model' items=78 subfolders=3 tab=txt2img
                         folders=['A:\\stable-diffusion-webui\\models\\Stable-diffusion',
                         'A:\\stable-diffusion-webui\\models\\Diffusers', 'models\\Reference'] list=0.05 thumb=0.01
                         desc=0.01 info=0.03 workers=4 sort=Default
14:07:52-491106 DEBUG    Networks: page='lora' items=32 subfolders=2 tab=txt2img
                         folders=['A:\\stable-diffusion-webui\\models\\Lora',
                         'A:\\stable-diffusion-webui\\models\\LyCORIS'] list=0.04 thumb=0.01 desc=0.01 info=0.05
                         workers=4 sort=Default
14:07:52-526113 DEBUG    Networks: page='style' items=288 subfolders=1 tab=txt2img
                         folders=['A:\\stable-diffusion-webui\\models\\styles', 'html'] list=0.05 thumb=0.00 desc=0.00
                         info=0.00 workers=4 sort=Default
14:07:52-534613 DEBUG    Networks: page='embedding' items=63 subfolders=1 tab=txt2img
                         folders=['A:\\stable-diffusion-webui\\embeddings'] list=0.06 thumb=0.01 desc=0.01 info=0.02
                         workers=4 sort=Default
14:07:52-537612 DEBUG    Networks: page='vae' items=2 subfolders=0 tab=txt2img
                         folders=['A:\\stable-diffusion-webui\\models\\VAE'] list=0.02 thumb=0.01 desc=0.00 info=0.00
                         workers=4 sort=Default
14:07:52-630625 DEBUG    UI initialize: img2img
14:07:52-746636 DEBUG    UI initialize: control models=A:\stable-diffusion-webui\models\control
14:07:53-234200 DEBUG    Read: file="ui-config.json" json=0 bytes=2 time=0.000
14:07:53-370711 DEBUG    UI themes available: type=Modern themes=32
14:07:53-532233 DEBUG    Reading failed: A:\sd.next\html\extensions.json [Errno 2] No such file or directory:
                         'A:\\sd.next\\html\\extensions.json'
14:07:53-533734 INFO     Extension list is empty: refresh required
14:07:53-884278 DEBUG    Extension list: processed=6 installed=6 enabled=6 disabled=0 visible=6 hidden=0
14:07:54-197817 DEBUG    Root paths: ['A:\\sd.next']
14:07:54-275824 INFO     Local URL: http://127.0.0.1:7860/
14:07:54-276824 DEBUG    Gradio functions: registered=1756
14:07:54-278824 DEBUG    FastAPI middleware: ['Middleware', 'Middleware']
14:07:54-281824 DEBUG    Creating API
14:07:54-467350 INFO     [AgentScheduler] Task queue is empty
14:07:54-468849 INFO     [AgentScheduler] Registering APIs
14:07:54-578861 DEBUG    Scripts setup: ['IP Adapters:0.029', 'AnimateDiff:0.011', 'Text-to-Video:0.006', 'X/Y/Z
                         Grid:0.014', 'Face:0.017', 'Image-to-Video:0.008', 'Stable Video Diffusion:0.006']
14:07:54-580363 DEBUG    Model metadata: file="metadata.json" no changes
14:07:54-581364 DEBUG    Torch mode: deterministic=False
14:07:54-582363 DEBUG    Desired Torch parameters: dtype=FP32 no-half=False no-half-vae=False upscast=False
14:07:54-583362 INFO     Setting Torch parameters: device=cuda dtype=torch.float32 vae=torch.float32 unet=torch.float32
                         context=no_grad fp16=None bf16=None optimization=Scaled-Dot-Product
14:07:54-585362 DEBUG    Model requested: fn=<lambda>
14:07:54-586863 INFO     Select: model="lustifySDXLNSFWSFW_v20 [34caf7f4fc]"
14:07:54-588362 DEBUG    Load model: existing=False
                         target=A:\stable-diffusion-webui\models\Stable-diffusion\lustifySDXLNSFWSFW_v20.safetensors
                         info=None
14:07:54-589862 DEBUG    Diffusers loading:
                         path="A:\stable-diffusion-webui\models\Stable-diffusion\lustifySDXLNSFWSFW_v20.safetensors"
14:07:54-590866 INFO     Autodetect: model="Stable Diffusion XL" class=StableDiffusionXLPipeline
                         file="A:\stable-diffusion-webui\models\Stable-diffusion\lustifySDXLNSFWSFW_v20.safetensors"
                         size=6617MB
Loading pipeline components... 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7/7  [ 0:00:06 < 0:00:00 , 1 C/s ]
14:08:01-589713 DEBUG    Setting model: pipeline=StableDiffusionXLPipeline config={'low_cpu_mem_usage': True,
                         'torch_dtype': torch.float32, 'load_connected_pipeline': True, 'extract_ema': False, 'config':
                         'configs/sdxl', 'use_safetensors': True, 'cache_dir':
                         'C:\\Users\\roy_p\\.cache\\huggingface\\hub'}
14:08:02-216305 INFO     Load embeddings: loaded=39 skipped=23 time=0.62
14:08:02-217804 DEBUG    Setting model: enable VAE slicing
14:08:02-244305 DEBUG    Read: file="A:\sd.next\configs\sdxl\vae\config.json" json=15 bytes=674 time=0.000
14:08:05-451730 DEBUG    GC: utilization={'gpu': 62, 'ram': 13, 'threshold': 80} gc={'collected': 10333, 'saved': 0.0}
                         before={'gpu': 14.89, 'ram': 4.03} after={'gpu': 14.89, 'ram': 4.03, 'retries': 0, 'oom': 0}
                         device=cuda fn=load_diffuser time=0.23
14:08:05-462730 INFO     Load model: time=10.63 load=7.00 embeddings=0.63 move=2.98 native=1024 {'ram': {'used': 4.03,
                         'total': 31.92}, 'gpu': {'used': 14.89, 'total': 23.99}, 'retries': 0, 'oom': 0}
14:08:05-465730 DEBUG    Script callback init time: system-info.py:app_started=0.06 task_scheduler.py:app_started=0.13
14:08:05-467230 INFO     Startup time: 21.08 torch=3.94 gradio=1.04 diffusers=0.40 libraries=1.67 extensions=0.61
                         face-restore=0.23 ui-en=0.26 ui-txt2img=0.07 ui-img2img=0.08 ui-control=0.14 ui-models=0.20
                         ui-settings=0.29 ui-extensions=0.40 ui-defaults=0.24 launch=0.13 api=0.11 app-started=0.19
                         checkpoint=10.89
14:08:05-469230 DEBUG    Save: file="config.json" json=35 bytes=1518 time=0.004
14:08:05-472230 INFO     Launching browser
14:08:07-499020 INFO     MOTD: N/A
14:08:09-349283 DEBUG    UI themes available: type=Modern themes=32
14:08:11-126032 INFO     Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64;
                         rv:129.0) Gecko/20100101 Firefox/129.0
14:08:28-955138 INFO     Select: model="flux_dev"
14:08:28-956637 DEBUG    Load model: existing=False
                         target=A:\stable-diffusion-webui\models\Stable-diffusion\flux_dev.safetensors info=None
14:08:33-074743 DEBUG    GC: utilization={'gpu': 62, 'ram': 44, 'threshold': 80} gc={'collected': 69, 'saved': 13.36}
                         before={'gpu': 14.89, 'ram': 14.2} after={'gpu': 1.53, 'ram': 14.2, 'retries': 0, 'oom': 0}
                         device=cuda fn=unload_model_weights time=0.3
14:08:33-077244 DEBUG    Unload weights model: {'ram': {'used': 14.2, 'total': 31.92}, 'gpu': {'used': 1.53, 'total':
                         23.99}, 'retries': 0, 'oom': 0}
14:08:33-089739 DEBUG    Diffusers loading:
                         path="A:\stable-diffusion-webui\models\Stable-diffusion\flux_dev.safetensors"
14:08:33-091239 INFO     Autodetect: model="Stable Diffusion" class=StableDiffusionPipeline
                         file="A:\stable-diffusion-webui\models\Stable-diffusion\flux_dev.safetensors" size=22700MB
Loading pipeline components...  33% ━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/6  [ 0:00:00 < -:--:-- , ? C/s ]
14:08:33-120755 ERROR    Diffusers failed loading:
                         model=A:\stable-diffusion-webui\models\Stable-diffusion\flux_dev.safetensors
                         pipeline=Autodetect/NoneType config={'low_cpu_mem_usage': True, 'torch_dtype': torch.float32,
                         'load_connected_pipeline': True, 'safety_checker': None, 'requires_safety_checker': False,
                         'local_files_only': False, 'extract_ema': False, 'config': 'configs/sd15', 'use_safetensors':
                         True, 'cache_dir': 'C:\\Users\\roy_p\\.cache\\huggingface\\hub'} Failed to load CLIPTextModel.
                         Weights for this component appear to be missing in the checkpoint.
                         Please load the component before passing it in as an argument to `from_single_file`.

                         text_encoder = CLIPTextModel.from_pretrained('...')
                         pipe = StableDiffusionPipeline.from_single_file(<checkpoint path>, text_encoder=text_encoder)

14:08:33-123755 ERROR    loading model=A:\stable-diffusion-webui\models\Stable-diffusion\flux_dev.safetensors
                         pipeline=Autodetect/NoneType: SingleFileComponentError
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ A:\sd.next\venv\lib\site-packages\diffusers\loaders\single_file.py:495 in from_single_file                           │
│                                                                                                                      │
│   494 │   │   │   │   try:                                                                                           │
│ ❱ 495 │   │   │   │   │   loaded_sub_model = load_single_file_sub_model(                                             │
│   496 │   │   │   │   │   │   library_name=library_name,                                                             │
│                                                                                                                      │
│ A:\sd.next\venv\lib\site-packages\diffusers\loaders\single_file.py:168 in load_single_file_sub_model                 │
│                                                                                                                      │
│   167 │   │   │   if not _is_model_weights_in_cached_folder(cached_model_config_path, name):                         │
│ ❱ 168 │   │   │   │   raise SingleFileComponentError(                                                                │
│   169 │   │   │   │   │   f"Failed to load {class_name}. Weights for this component appear to be missing in the chec │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
SingleFileComponentError: Failed to load CLIPTextModel. Weights for this component appear to be missing in the checkpoint.

During handling of the above exception, another exception occurred:

╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ A:\sd.next\modules\sd_models.py:1222 in load_diffuser                                                                │
│                                                                                                                      │
│   1221 │   │   │   │   │   │   sd_hijack_accelerate.restore_accelerate()                                             │
│ ❱ 1222 │   │   │   │   │   sd_model = pipeline.from_single_file(checkpoint_info.path, **diffusers_load_config)       │
│   1223 │   │   │   │   │   # sd_model = patch_diffuser_config(sd_model, checkpoint_info.path)                        │
│                                                                                                                      │
│ A:\sd.next\venv\lib\site-packages\huggingface_hub\utils\_validators.py:114 in _inner_fn                              │
│                                                                                                                      │
│   113 │   │                                                                                                          │
│ ❱ 114 │   │   return fn(*args, **kwargs)                                                                             │
│   115                                                                                                                │
│                                                                                                                      │
│ A:\sd.next\venv\lib\site-packages\diffusers\loaders\single_file.py:510 in from_single_file                           │
│                                                                                                                      │
│   509 │   │   │   │   except SingleFileComponentError as e:                                                          │
│ ❱ 510 │   │   │   │   │   raise SingleFileComponentError(                                                            │
│   511 │   │   │   │   │   │   (                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
SingleFileComponentError: Failed to load CLIPTextModel. Weights for this component appear to be missing in the checkpoint.
Please load the component before passing it in as an argument to `from_single_file`.

text_encoder = CLIPTextModel.from_pretrained('...')
pipe = StableDiffusionPipeline.from_single_file(<checkpoint path>, text_encoder=text_encoder)

14:08:33-342294 DEBUG    Setting changed: sd_model_checkpoint=flux_dev progress=True
14:08:33-343793 DEBUG    Save: file="config.json" json=35 bytes=1491 time=0.001
14:10:00-453632 DEBUG    Server: alive=True jobs=1 requests=42 uptime=130 memory=14.14/31.92 backend=Backend.DIFFUSERS
                         state=idle
Backend

Diffusers
UI

ModernUI
Branch

Master
Model

Other
Acknowledgements

[X] I have read the above and searched for existing issues
[X] I confirm that this is classified correctly and its not an extension issue
vladmandic / automatic
