Trajectory Consistency Distillation Support

[jesenzhang]

Trajectory Consistency Distillation needs a confyui native TCDScheduler https://github.com/jabir-zheng/TCD https://huggingface.co/h1t/TCD-SDXL-LoRA

[dfl]

I have attempted this but need a little help finishing it! https://github.com/jabir-zheng/TCD/issues/7

[comfyanonymous]

I'm getting some success sampling this with euler ancestral while adjusting the eta of the sampler. The workflow should be attached to the image.

[jesenzhang]

I'm getting some success sampling this with euler ancestral while adjusting the eta of the sampler. The workflow should be attached to the image.

I have already tried it，and it would not work with '--force-fp16'.

[MoonRide303]

Workflow that works pretty fine for me (with SDXL-Lightning variant):

MoonRide SDXL-Lightning+TCD workflow v1.json

[lucianosb]

Workflow that works pretty fine for me (with SDXL-Lightning variant):

MoonRide SDXL-Lightning+TCD workflow v1.json

I also tested this with my own base SDXL model merge. It worked fine!

I just couldn't make it work with the DPMPP_3M_SDE Sampler, I'll keep exploring.

Thank you so much!

[wangfeng35]

tcd is awesome! Hope it will be supported.

[dfl]

I have this working pretty well with a modified Euler A sampler: https://github.com/dfl/comfyui-tcd-scheduler

however AFAICT the reference diffusers implementation uses modified DDPM. I haven't yet figured out how to properly rescale the samples with that though. https://github.com/dfl/comfyui-tcd-scheduler/issues/3

Also:

In the diffusers implementation I get these values for 8 steps before they are modified by gamma: timesteps: [999, 879, 759, 639, 499, 379, 259, 139] sigmas: [0.9977, 0.9912, 0.9735, 0.9349, 0.8499, 0.7359, 0.5840, 0.3961]

However in comfy the timesteps are slightly different... is this to be expected due to minor differences in implementation frameworks? [... 876, 751, 626, 501, 376, 251, 126, 1]

[dfl]

someone asked about an img2img workflow... are there any extant nodes to extend this for img2img without rewriting the native KSampler?

Also opened up the question of how TCD might ever be natively integrated. Given my current understanding of the ComyfUI architecture, I don't see how sending additional KSampler parameters like TCD's gamma/eta could be possible... @comfyanonymous what are your thoughts/advice on this please? https://github.com/dfl/comfyui-tcd-scheduler/issues/4#issuecomment-2020088752

I think the general notion of blending between stochastic and deterministic is pretty cool, and could be useful even without the TCD LoRA, just with more typical/traditional sampling steps.

[let-me-cook]

New Hyper-SD (Trajectory Segmented Consistency Distillation) from bytedance apparently runs atop of diffusers's TCDScheduler aswell. https://hyper-sd.github.io/

[YanzuoLu]

New Hyper-SD (Trajectory Segmented Consistency Distillation) from bytedance apparently runs atop of diffusers's TCDScheduler aswell. https://hyper-sd.github.io/

Yes we adopt the TCDScheduler on our unified LoRAs. And we find the corresponding implementation from @dfl in ComfyUI does not generate proper outcomes when step size is 1 or 2. https://github.com/dfl/comfyui-tcd-scheduler Can anyone help? We are the AutoML team from ByteDance and want to integrate our unified LoRAs into ComfyUI and provide more workflows. Thanks a lot! @lucianosb @wangfeng35 @jesenzhang @let-me-cook @dfl

[let-me-cook]

Not a dev so i will tag comfy instead @comfyanonymous, sir comfy please do the needful sirs

[hahy36]

@comfyanonymous I also have the same need. Althought third-party plugins can achieve some of the functions of TCD ,but they lack the 'denoise'.

[dfl]

Native KSampler does not support extra parameters like TCD’s Gamma.

[JettHu]

I've implemented another tcd comfyui node. repo: https://github.com/JettHu/ComfyUI-TCD

[YanzuoLu]

@JettHu Nice done, friend! We have tested it on our Hyper-SD and it works perfect!!! We will soon update it on our HF page. Thanks for your great job!

[JettHu]

@YanzuoLu I am preparing to submit a PR for comfyui to add tcd to the comfyui official scheduler.

[JettHu]

Because the default KSampler(KSamplerAdvanced) lacks the eta parameter, the current implementation has to use SamplerCustom (otherwise the default 0.3 is used) to see if the author is willing to add an additional eta parameter.

[JettHu]

@JettHu Nice done, friend! We have tested it on our Hyper-SD and it works perfect!!! We will soon update it on our HF page. Thanks for your great job!

So does Hyper-SD also need to use tcd scheduler? I am happy to provide some help.

[YanzuoLu]

@JettHu Yes, our 1-step unified lora is able to inference on 1-8 steps with the help of TCDScheduler within single checkpoint! And we found previous implementation of it behaves differently from diffusers and your implementation works great! Thanks for your help!

[bigmover]

@JettHu Yes, our 1-step unified lora is able to inference on 1-8 steps with the help of TCDScheduler within single checkpoint! And we found previous implementation of it behaves differently from diffusers and your implementation works great! Thanks for your help!

Are you working on supporting Hyper-SD? It's available for use with Diffusers or ComfuUI or WebUI?

[MoonRide303]

@JettHu Yes, our 1-step unified lora is able to inference on 1-8 steps with the help of TCDScheduler within single checkpoint! And we found previous implementation of it behaves differently from diffusers and your implementation works great! Thanks for your help!

Are you working on supporting Hyper-SD? It's available for use with Diffusers or ComfuUI or WebUI?

Hyper-SD models work pretty well with Euler / DDIM sampler and sgm_uniform scheduler, as in their reference workflows. You might just need to reduce LoRA strength a bit, to let's say 0.8:

Hyper-SDXL workflow.json

Only 1-step Hyper-SDXL Unet needs their specific scheduler - and they've provided ComfyUI custom node for that in their repo, already.

And TCD works pretty fine with Euler Ancestral and sgm_uniform sampler:

TCD-SDXL workflow.json