search

nod-ai / SHARK-Studio

SHARK Studio -- Web UI for SHARK+IREE High Performance Machine Learning Distribution
Apache License 2.0
1.42k stars 171 forks source link

Clip Skip for SD Txt2Img #1865

Open njsharpe opened 1 year ago

njsharpe commented 1 year ago

Seeing the completion of diffusers#3212 (which was actually implemented in transformers, but whatever), it should be easier to implement clip skip as a feature into SHARK.

However, clip skip is intended to be a "dynamic" variable which, if implemented in the compilation-based SHARK, may require a re-compile every time this variable changes, which is not ideal. If it can be implemented without a re-compile every time the variable changes, the implementation itself should be dead easy in the model wrapper, and would look something like this:

self.text_encoder = CLIPTextModel.from_pretrained(model_id, subfolder="text_encoder")
# LoRA Checking Here...
layers = self.text_encoder.text_model.encoder.layers
if clip_skip > 0:
    self.text_encoder.text_model.encoder.layers = layers[:-clip_skip]

Note here that the clip_skip arg is fed by the user, and would likely come from args.clip_skip, where we store all the other user-defined args from the UI.

If my knowledge of this pipeline & SHARK were any better, I'd do this myself and just submit a PR but here we are.

monorimet commented 1 year ago

Dynamic is possible (see Chatbot/LLM implementations) but implementation varies among models. I think this is worth tracking. @gpetters94 do you have any thoughts on this?