[!NOTE] Officially implemented here: https://github.com/TencentQQGYLab/ComfyUI-ELLA

ComfyUI_ELLA

ComfyUI Implementaion of ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

[!NOTE]
As per the ELLA developers / researchers, only the SD 1.5 checkpoint is released.

Installation

To install, simply navigate to custom_nodes, and inside that directory, do git clone https://github.com/ExponentialML/ComfyUI_ELLA.git

These models must be placed in the corresponding directories under models.

Example: ComfyUI/models/ella/model_file

The directories should be created automatically. If they are not, please create them manually.

Place the ELLA Model under a new folder ella: https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors
Create a folder called t5_model. Navigate to that folder (you must be in that directory), and git clone https://huggingface.co/google/flan-t5-xl to download the t5 model.
If you don't wish to use git, you can dowload each indvididually file manually by creating a folder t5_model/flan-t5-xl, then download every file from here, although I recommend git as it's easier.

In summary, you should have the following model directory structure:

To get started quickly, a workflow is provided in the workflow directory.

ella: The loaded model using the ELLA Loader.
text: Conditioning prompt. All weighting and such should be 1:1 with all condiioning nodes.
sigma: The required sigma for the prompt. It must be the same as the KSampler settings. Without the workflow, initially this will be a float. You can simply right click the node, convert sigma to input, then use the Get Sigma node.

All conditioning nodes should be supported, as well as prompt weighting and ControlNet.

Thanks to the following for open sourcing. Please follow their respective licensing.