sdxl with .and() with weights?

[rb-synth]

Is it possible to use the .and() format with weights with SDXL? I'm trying the following:

from compel import Compel, ReturnedEmbeddingsType
from diffusers import DiffusionPipeline
import torch

pipeline = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", 
    variant="fp16", 
    use_safetensors=True, 
    torch_dtype=torch.float16,
).to("cuda")
compel = Compel(
    tokenizer=[pipeline.tokenizer, pipeline.tokenizer_2] ,
    text_encoder=[pipeline.text_encoder, pipeline.text_encoder_2], 
    returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED, 
    requires_pooled=[False, True],
)
prompt = '("prompt part 1", "prompt part 2").and(0.5, 0.5)'
conditioning, pooled = compel(prompt)

The error is:

Traceback (most recent call last):
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/richardbrown/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/home/richardbrown/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/richardbrown/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/richardbrown/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/home/richardbrown/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/richardbrown/.vscode-server/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "/home/richardbrown/Documents/Code/train_inpaint/quick.py", line 9, in <module>
    conditioning, pooled = compel(prompt)
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/site-packages/compel/compel.py", line 135, in __call__
    output = self.build_conditioning_tensor(text_input)
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/site-packages/compel/compel.py", line 112, in build_conditioning_tensor
    conditioning, _ = self.build_conditioning_tensor_for_conjunction(conjunction)
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/site-packages/compel/compel.py", line 192, in build_conditioning_tensor_for_conjunction
    [padded_empty_conditioning, _] = self.pad_conditioning_tensors_to_same_length([empty_conditioning, this_conditioning])
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/site-packages/compel/compel.py", line 270, in pad_conditioning_tensors_to_same_length
    return type(self)._pad_conditioning_tensors_to_same_length(conditionings, emptystring_conditioning=emptystring_conditioning)
  File "/home/richardbrown/mambaforge/envs/inpaint_dreambooth/lib/python3.10/site-packages/compel/compel.py", line 225, in _pad_conditioning_tensors_to_same_length
    c0_shape = conditionings[0].shape
AttributeError: 'tuple' object has no attribute 'shape'

[damian0815]

uh. it should be technically possible but i lack the hardware and time to fix SDXL issues. internally .and() just concatenates the embeddings so you could maybe try doing that by hand.

[rb-synth]

Thanks, so I would compile each one individually and combine the conditioning results. But what about the pooled results, would that be the same?

[damian0815]

yes, it should work the same way

[rndm-jpg]

Hey @rb-synth @damian0815 been having the same issue, too. Do any of you have a sample code to follow regarding how to fix weighting + conjunction on SDXL?

Would be extremely helpful for us 🙏

[kno10]

Proposed fix: #88