[bug]: Local concepts (embeddings) are not taken into account for not diffuser models

[i3oc9i]

Is there an existing issue for this?

• [X] I have searched the existing issues

OS

macOS

GPU

mps

VRAM

128GB

What version did you experience this issue on?

3.0.0+a0 - commit(e9c2f173)

What happened?

Concepts (embeddings) are not taken into account when a non diffuser model is selected.

In the screenshoot you can observe than when the sd-2.1-768 stock model is used, then the local concepts included in the prompt are taken into account,

>> Loaded local embedding for trigger wer
>> Loaded local embedding for trigger nfixer-271
>> Loaded local embedding for trigger nfixernext
>> Preparing tokens for textual inversion nfixernext...
>> Preparing tokens for textual inversion wer...
>> Preparing tokens for textual inversion nfixer-271...

later, when the x-rmada-merge-sd21 model is selected, it is converted on-fly as a diffuser model, but then the concepts included in the prompt are not recognised as valid one.

This concept is not a local embedding trigger, nor is it a HuggingFace concept. Generation will continue without the concept.

Screenshots

(sd-15) invoke> !switch sd-2.1-768
>> Current VRAM usage:  0.00G
>> Offloading sd-15 to CPU
>> Loading diffusers model from stabilityai/stable-diffusion-2-1
  | Using more accurate float32 precision
Fetching 13 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 40782.31it/s]
  | Default image dimensions = 768 x 768
>> Loading embeddings from /Users/ivano/Code/Ai/@Stuffs/invokeai.models/embeddings
** Notice: others/bad_prompt.pt was trained on a model with an incompatible token dimension: 1024 vs 768.
>> Textual inversion triggers: nfixer-271, nfixernext, wer
>> Model loaded in 3.33s
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
(sd-2.1-768) invoke> [ <wer> <nfixer-271> <nfixernext> ] a dog in the garden
>> Patchmatch initialized
>> Loaded local embedding for trigger wer
>> Loaded local embedding for trigger nfixer-271
>> Loaded local embedding for trigger nfixernext
>> Preparing tokens for textual inversion nfixernext...
>> Preparing tokens for textual inversion wer...
>> Preparing tokens for textual inversion nfixer-271...

..... (missing lines)

(sd-2.1-768) invoke> !switch x-rmada-merge-sd21
>> Current VRAM usage:  0.00G
>> Cache limit (max=2) reached. Purging sd-15
>> Offloading sd-2.1-768 to CPU
>> Loading x-rmada-merge-sd21 from /users/ivano/junk/sd/rmadamerge-v5.0.safetensors
>> Converting legacy checkpoint x-rmada-merge-sd21 into a diffusers model...
>> Offloading sd-2.1-768 to CPU
** Custom VAE "None" found, but corresponding diffusers model unknown
** Using "stabilityai/sd-vae-ft-mse"; If this isn't right, please edit the model config
  | Loading diffusers VAE from stabilityai/sd-vae-ft-mse
  | Using more accurate float32 precision
  | global_step key not found in model
  | Using external VAE specified in config
>> Model loaded in 7.36s
>> Setting Sampler to k_lms (LMSDiscreteScheduler)
(x-rmada-merge-sd21) invoke> [ <wer> <nfixer-271> <nfixernext> ] a dog in the garden
This concept is not a local embedding trigger, nor is it a HuggingFace concept. Generation will continue without the concept.
This concept is not a local embedding trigger, nor is it a HuggingFace concept. Generation will continue without the concept.
This concept is not a local embedding trigger, nor is it a HuggingFace concept. Generation will continue without the concept.

....

Additional context

Link to x-rmada-merge-sd21 model on CIVITAI

Contact Details

No response

[lstein]

I’ll investigate

[n00mkrad]

I’ll investigate

Also, please improve the CLI messages for the embeddings. The error messages do not print the names. This makes troubleshooting very annoying if you use multiple embeddings.

Here are my proposed changes for concepts_lib.py: "This concept is not a local embedding trigger, nor is it a HuggingFace concept. Generation will continue without the concept." change to: f"{concept_name} is not a local embedding trigger, nor is it a HuggingFace concept. Generation will continue without the concept."

">> Invalid embedding format" change to: f">> Invalid embedding format: {os.path.basename(embedding_file)}"

[lstein]

Sorry for the delay. I've been on vacation for the last week and not getting much time to code. I'll get this figured out and fixed sometime in the coming week.

[lstein]

Bug found and fixed. Will appear as soon as the PR is reviewed.

[i3oc9i]

Thank you so much