search

jabir-zheng / TCD

Official Repository of the paper "Trajectory Consistency Distillation"
https://mhh0318.github.io/tcd
290 stars 14 forks source link

implementation in ComfyUI framework #7

Closed dfl closed 2 months ago

dfl commented 4 months ago

Thank for your excellent contribution!

I am attempting to implement this for the ComfyUI community, however there are some differences in structure between Diffusers (more explicit time scheduling) and ComfyUI (array of sigmas, and abstracted scheduling).

The tricky part is how to do the adjusted timestep calculation using eta/gamma. Something is incorrect so the image just goes black after the first step, when eta is > 0.

I wonder someone on your team would be able to look at my code please? (or anyone else here on GitHub who is good with diffusers and comfyui) https://github.com/dfl/comfyui-tcd-scheduler

jabir-zheng commented 4 months ago

Thank you sincerely for your tremendous efforts!

We will attempt to address this during our available time. However, as we are not well-versed in ComfyUI, this process might take some time.

Assistance from individuals experienced with ComfyUI would be greatly appreciated!

wandrzej commented 4 months ago

image top batch of 4 - sdxl checkpoint with 16 steps middle - same checkpoint with 4 steps bottom, same checkpoint with 4 steps and this LoRA applied. So looks like even without special implementation this gives far more crisp results on it's own, through the vanilla Load LoRA node.

mhh0318 commented 4 months ago

image top batch of 4 - sdxl checkpoint with 16 steps middle - same checkpoint with 4 steps bottom, same checkpoint with 4 steps and this LoRA applied. So looks like even without special implementation this gives far more crisp results on it's own, through the vanilla Load LoRA node.

Yup. It shares the same scheduler with LCM when gamma=0. When using other samplers, you could also find an alignment. Could you provide which type of sampler used for the bottom batch?

wandrzej commented 4 months ago

Yup. It shares the same scheduler with LCM when gamma=0. When using other samplers, you could also find an alignment. Could you provide which type of sampler used for the bottom batch?

99% sure it was DPM++ SDE, with CFG ~2, as those were the settings I was using with a lightning version of a checkpoint. Unlike the Lightning LoRA which requires euler + sgm scheduler, the DPM++ SDE seems to be the consensus preferred option. I tried it without much hope of it working, and it just did. Most likely not the right way of doing it, nor optimal, but the effect is clear.

I'll later re-run the experiment to confirm.

dfl commented 4 months ago

@jabir-zheng please see my comments here... can you assist in my understanding? https://github.com/dfl/comfyui-tcd-scheduler/issues/1#issuecomment-2009312833

dfl commented 4 months ago

I have created a ddpm branch in my repo, but I need some help with injecting the gamma-modified sigma https://github.com/dfl/comfyui-tcd-scheduler/tree/ddpm for comparison, there is a modified EulerA sampler from the main branch.

mhh0318 commented 2 months ago

Please check this awesome implementation. https://github.com/jabir-zheng/TCD/issues/16 by @JettHu

bigmover commented 4 weeks ago

Please check this awesome implementation. #16 by @JettHu

Does this implementation lack a schedule or I missed something?