[bug]: If I put controlnet for sdxl here InvokeAI\models\sdxl\controlnet - Invoke is broken

[mirek190]

Is there an existing issue for this?

• [X] I have searched the existing issues

OS

Windows

GPU

cuda

VRAM

24

What version did you experience this issue on?

3.2-rc3

What happened?

If I put controlnet for sdxl here InvokeAI\models\sdxl\controlnet - Invoke is broken. Until I remove those controlnet SDXL models then works again

control-lora-canny-rank256.safetensors control-lora-depth-rank256.safetensors control-lora-recolor-rank256.safetensors control-lora-sketch-rank256.safetensors

Desired action:
1. Generate images with the browser-based interface
2. Explore InvokeAI nodes using a command-line interface
3. Run textual inversion training
4. Merge models (diffusers type only)
5. Download and install models
6. Change InvokeAI startup options
7. Re-run the configure script to fix a broken install or to complete a major upgrade
8. Open the developer console
9. Update InvokeAI
10. Run the InvokeAI image database maintenance script
11. Command-line help
Q - Quit
Please enter 1-11, Q: [1] 1
Starting the InvokeAI browser-based UI..
[2023-09-29 21:56:04,639]::[InvokeAI]::INFO --> Patchmatch initialized
E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\torchvision\transforms\functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
[2023-09-29 21:56:07,315]::[uvicorn.error]::INFO --> Started server process [104836]
[2023-09-29 21:56:07,315]::[uvicorn.error]::INFO --> Waiting for application startup.
[2023-09-29 21:56:07,315]::[InvokeAI]::INFO --> InvokeAI version 3.2.0rc3
[2023-09-29 21:56:07,315]::[InvokeAI]::INFO --> Root directory = E:\DIFFUSION\InvokeAI
[2023-09-29 21:56:07,316]::[InvokeAI]::INFO --> Using database at E:\DIFFUSION\InvokeAI\databases\invokeai.db
[2023-09-29 21:56:07,317]::[InvokeAI]::INFO --> GPU device = cuda NVIDIA GeForce RTX 3090
[2023-09-29 21:56:07,334]::[InvokeAI]::INFO --> Scanning E:\DIFFUSION\InvokeAI\models for new models
[2023-09-29 21:56:07,340]::[uvicorn.error]::ERROR --> Traceback (most recent call last):
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\starlette\routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\starlette\routing.py", line 566, in __aenter__
    await self._router.startup()
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\starlette\routing.py", line 648, in startup
    await handler()
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\app\api_app.py", line 78, in startup_event
    ApiDependencies.initialize(config=app_config, event_handler_id=event_handler_id, logger=logger)
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\app\api\dependencies.py", line 131, in initialize
    model_manager=ModelManagerService(config, logger),
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\app\services\model_manager_service.py", line 341, in __init__
    self.mgr = ModelManager(
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\backend\model_management\model_manager.py", line 354, in __init__
    self._read_models(config)
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\backend\model_management\model_manager.py", line 377, in _read_models
    self.scan_models_directory()
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\backend\model_management\model_manager.py", line 997, in scan_models_directory
    model_config: ModelConfigBase = model_class.probe_config(
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\backend\model_management\models\stable_diffusion.py", line 57, in probe_config
    model_format = cls.detect_format(path)
  File "E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\backend\model_management\models\stable_diffusion.py", line 106, in detect_format
    raise ModelNotFoundException()
invokeai.backend.model_management.models.base.ModelNotFoundException

[2023-09-29 21:56:07,341]::[uvicorn.error]::ERROR --> Application startup failed. Exiting.
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<FastAPIEventService.__dispatch_from_queue() running at E:\DIFFUSION\InvokeAI\.venv\lib\site-packages\invokeai\app\api\events.py:48> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Desired action:
1. Generate images with the browser-based interface
2. Explore InvokeAI nodes using a command-line interface
3. Run textual inversion training
4. Merge models (diffusers type only)
5. Download and install models
6. Change InvokeAI startup options
7. Re-run the configure script to fix a broken install or to complete a major upgrade
8. Open the developer console
9. Update InvokeAI
10. Run the InvokeAI image database maintenance script
11. Command-line help
Q - Quit
Please enter 1-11, Q: [1]

Screenshots

No response

Additional context

No response

Contact Details

No response

[Millu]

InvokeAI only supports use of diffusion format ControlNets - for example, this is a canny ControlNet you'll want to use:

https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0

You can download it through the model manager with the repo id: diffusers/controlnet-canny-sdxl-1.0

[TheTyrius]

On a related note: Is there a reason SDXL ControlNets have not been added to the invokeai/configs/INITIAL_MODELS.yaml alongside those provided for sd-1? As a user this "pre-selection" provides a lot of value as a shortcut through the "model-jungle" :)

[lstein]

control-lora-canny-rank256.safetensors

If you provide me with a list of SDXL controlnets that should be offered, along with short names to describe them, I'd be happy to add them to INITIAL_MODELS.yaml.

[Jaid]

@lstein Is this project compatible with Invoke in its current state?

[TheTyrius]

If you provide me with a list of SDXL controlnets that should be offered, along with short names to describe them, I'd be happy to add them to INITIAL_MODELS.yaml.

Sadly I am not very knowledgeable when it comes to model formats and compatibility. I tend to be wary of "random" repos (maybe unwarranted these days?) So to me the sdxl controlnet collection by the diffusers org on HF looks promising: https://huggingface.co/collections/diffusers/sdxl-controlnets-64f9c35846f3f06f5abe351f They seem to be SDXL equivalents to canny/depth/zoe-depth models we've had for sd1.5 (so could do with similar descriptions I guess?).

By my crude understanding these come in .safetensors format (no subfolders) with json config (your PR mentions .yaml is that more common?).

I guess the "full" versions are most relevant, as I would likely try T2I adapters if I need smaller size (but again, your judgement is likely 100x more useful on this).

Thanks for taking the time, as I said I highly value the "curation" of a know-good model list by people more knowledgeable than myself :)

[psychedelicious]

The model manager was rewritten for v4. Model installs now happen via the UI, not putting files in folders. Please create a new issue if the problem persisted with the latest release (v4.1.0).