XavierXiao / Dreambooth-Stable-Diffusion

Implementation of Dreambooth (https://arxiv.org/abs/2208.12242) with Stable Diffusion
MIT License
7.6k stars 795 forks source link

"Sks" is a bad identifier. It's a gun. #71

Closed exo-pla-net closed 5 months ago

exo-pla-net commented 2 years ago

Using sks, you'll get guns in your generations.

Here's how the original DreamBooth authors found good identifiers:

In a nutshell, our approach is to find relatively rare tokens in the vocabulary, and
then invert these rare tokens into text space. In order to do this, we first perform a rare-token lookup in the
vocabulary and obtain a sequence of rare token identifiers f(Vˆ ), where f is a tokenizer; a function that maps
character sequences to tokens and Vˆ is the decoded text stemming from the tokens f(Vˆ ). This sequence can
be of variable length k with k being a hyperparameter of our method. We find that relatively short sequences
of k = {1, ..., 3} work well. Then, by inverting the vocabulary using the de-tokenizer on f(Vˆ ) we obtain a
sequence of characters that define our unique identifier Vˆ . We observe that using uniform random sampling
without replacement of tokens that correspond to 3 or fewer Unicode characters (without spaces) and using
tokens in the T5-XXL tokenizer range of {5000, ..., 10000} works well.

The DreamBooth authors unfortunately declined to provide what good identifiers they found using their technique. "Sks", I understand, was selected without much scrutiny.

Instead of settling for the problematic "sks", we should instead replicate what the DreamBooth authors did (or maybe just ask them), to find a genuinely good unique identifier.

I lack the domain knowledge to do this myself.

enzyme69 commented 1 year ago

A bit OT, but can we have multiple persons in one model? like sks 1 sks 2 sks 3?

rydogjones commented 1 year ago

No wonder I kept getting guns in my pics! :-) I use t@y now and it seem to do the trick.

dcsan commented 1 year ago

@enzyme69 thats about multiconcept training. how far did you get with that?

rydogjones commented 1 year ago

Could never get multiple trainings working well, meaning train on one, then add another training to the same model, etc... But I did get training on multiple people like Couples in a single training. You can't specify the people individually but it knows who is a girl and who is a boy and most images include both, some include one or the other and some include multiples of each. Nothing special in the process except good couple training pics.

Vizzana commented 5 months ago

what about sks? why you use Sks as identifier? i think Sks is too similar to sks. Maybe that is language swift?