search

p1atdev / LECO

Low-rank adaptation for Erasing COncepts from diffusion models.
https://arxiv.org/abs/2303.07345
Apache License 2.0
307 stars 23 forks source link

Penultimate CLIP layer #11

Closed catboxanon closed 1 year ago

catboxanon commented 1 year ago

AFAIK, Waifu Diffusion is trained using the penultimate CLIP layer, just as NovelAI did. Currently there isn't a config option for that. Shouldn't there be one? I would imagine that would improve the results for the WD1.5 examples.

p1atdev commented 1 year ago

Hi,

SDv2.x models always use the penultimate layer, so it is no problem for SDv2.x like WD1.5. (see https://github.com/huggingface/diffusers/issues/1721#issuecomment-1372737379)

But some v1.x models prefer CLIP skip 2, so I'll add an option for it.

Thanks for pointing out.

catboxanon commented 1 year ago

Looks like this was implemented as part of https://github.com/p1atdev/LECO/pull/13 so I'm gonna close this.