Struggling to get any likeness after training model

[Antoinevdlb]

Hey everyone!

I trained a model today using pictures of my girlfriend (with consent of course). Used the recommended 52 "Instances" images (mix of closeup portraits, waist shots and full shots) + 200 "Class" Images of women in their 20's all pulled from Unsplash. All the pictures were cropped to 1:1 and highlighted varied poses and lighting composition.

Settings I ran with her Subject_Name = Person, Instance_Name= sks, 2500 steps

Examples of the training set

Instances Pictures:

Class Pictures:

--

I've been really disappointed in the results, especially when comparing them to Astraea (which I believe uses Stable Diffusion and Dream-Booth as well, and was trained on the same set of instances. Am I doing something wrong?

--

Examples of Renders:

Render 1

Astraea

VS

Fast SD

--

Render 2

Astraea

VS

Fast SD

--

Render 3

Fast SD

VS

Astraea

--

In general, the art through SD Repo is interesting, but you'll notice the likeness is completely off. I use the "sks person" trigger in both prompts and they are similar in every-way (including CFG scale and number of render steps). I know the training was successful because if I type only "sks person" in SD as a prompt I get a picture of the model it was trained on.

Anyone have any thoughts on why the results are so poor?

[jujpenabe]

I get better results with more CLASS images, between 300 and 400 images (I'm now testing with 1000+). In your case, try using the subject word "woman". I think the number of steps is fine, I have seen people with good results with 1000 steps or less, although I am trying with more steps, something between 2500-4000.

To speed up the loading/generation of the class images, you can try downloading some regularization images and saving them somewhere in gdrive, I use This

[janekm]

I'd suggest just training with class images that were generated by SD as opposed to real-world ones. (like the person_ddim ones). The class images are supposed to (to a point) preserve what SD would normally generate with your class.

[TheLastBen]

You flipped class pictures with instance pictures, try again with the opposite.

[Antoinevdlb]

You flipped class pictures with instance pictures, try again with the opposite.

They were miss labeled only on this thread. I ran them correctly on the collab. Maybe I need a lot more than 52 class pictures like someone suggested?

[TheLastBen]

@Antoinevdlb the CLASS pictures must be 200+ and instance pictures 50 is fine, try unchecking the fp16 option and rerun the training. here is an example of good results : https://www.reddit.com/r/StableDiffusion/comments/xxh5y6/dreambooth_consistently_blows_me_away_results/

[Antoinevdlb]

lol i keep getting this wrong sorry...

I definitely have 52 pictures of my GF's face (Instance) and 200 pictures of young women (Class)

When I ran it last time I ran it with fp16 UNCHECKED

Could I need more class pictures? Should I focus only on sharp portraits and forget full body shots? I also worry the sks person trigger doesn't work great potentially

[Antoinevdlb]

I'll try again, using the full set of 100 pictures I have for instance and the 1000 I have for class. We'll see how it goes.

I have the ability to run this locally (Titan RTX card). Would there be any benefit to that?

[TheLastBen]

don't use sks person, use "young woman" as a CLASS name, and any other identifier as an instance

[TheLastBen]

The important thing is the code, as long as google colab can run the script, you don't need a different hardware

[Antoinevdlb]

don't use sks person, use "young woman" as a CLASS name, and any other identifier as an instance

In this case, just to make sure i have this right:

Subject name would be "Young Women" and instance name "Jennie" for example?

[TheLastBen]

Yes, but for the instance, it's better to add something to the name eg : "Jennieabcd" to avoid conflict with the Jennies SD already knows

[Antoinevdlb]

using sks has been making me laugh, the model keeps trying to stick sks guns everywhere lol.

[TheLastBen]

exactly