preview samples show good likeness but ckpt file shows bad one ?

[1blackbar]

Describe the bug

Results from ckpt vs sample images during training is no the same trained for 5000k iterations, sampels looked very good with perfec likeness on close head shots and mid shots since 3500, but when converted 4500 to ckpt the likeness is pretty bad even on close head shots , kinda looks like 500 steps not 4500 ? Not sure whats going on, maybe saving multiple times like that messes up training .

Also worth to note i did lot of trainings yuesterday with 15 imgs, and had MUCH better resutls with likeness even with 800 steps and 15 imgs than with 4500 15imgs and todays code with saving checkpoint every 500 steps.There must be something in the code that is degrading results, maybe not everything is merged as it should. Its not overtrained, its undertrained somehow after ckpt conversion.Overtrained would do great likeness but pretty much only training images or artifacted images, this is not it, in case there will be questions about it.

Reproduction

train for 5k steps with 15 imgs and export to ckpt when sampe imgs are very good, test ckpt in webui - results in pretty bad likeness that looks like much lower steps.

Logs

No response

System Info

colab

[shadowlocked]

Can confirm this from a model trained today.

[1blackbar]

yeah we should compare 1500steps trained without saving in middle vs saving every 500 steps till 1500 and test the 1500 ones from both sessions... but there is something going on there also my prompt when i test just 3 letter instsance word withou class it does random stuff like extra glasses for no reason, and with class and 3 letter word it does random men mixed with instance 3 word subject... so for now id use yesterdays repo

[shadowlocked]

Yesterday's repo is what I am using, but it doesn't function like yesterday, because the trained model gets saved into sub-folders in the stated trained directory. One is called 0 and the next something like 4000 (based on steps). For now I am going to try moving the actual files to where they should be. I solved it earlier by changing the paths to the new unexpected sub-folder.

[ShivamShrirao]

May be wrong sub folder is being used for ckpt conversion. Can u verify the latest steps are being used ?

[zakinp]

i got the opposite results here, lmao. The preview images are horrible but the results in webui are looking good. It seems related to original ckpt model, iterations, learning rate, "--train_text_encoder", and settings in webui

[shadowlocked]

May be wrong sub folder is being used for ckpt conversion. Can u verify the latest steps are being used ?

Correct folder is being used. The backend code is now saving folder-named checkpoints inside the specified save directory. With a max step of 10k and an actual step of 4k, this guarantees at least two sub-folders will be created - '0' and '4000' (or whatever your specified steps were). This breaks the script.

[ShivamShrirao]

There is a next cell after training which assigns the WEIGHTS_DIR variable with the latest weights path. Max steps one.

[shadowlocked]

There is a next cell after training which assigns the WEIGHTS_DIR variable with the latest weights path. Max steps one.

Okay. Might be worth including some text to explain what is going on to people at that point, it took me a while to figure out, and I could not understand why the WEIGHTS_DIR box was at the very end.

[andreae293]

i too noticed heavy distorsion and artifact the more i trained, is there a way to avoid those artifacts?

[ShivamShrirao]

@andreae293 do not train more. It overfits.

[andreae293]

but is this overfitting? the face gets totally distorted and does not resemble the original one at all those artifacts for me starts after 2000 steps not sure if this is related to this issue

[1blackbar]

Well this topic is old news, im gettin great results now , using saving checkpoints and usually its there at 2400-2800 steps and 20 images, it even works with 3 images and 800 steps but its not that posable (harder to get different head angles if theyre not in face set for learning) the sample previews are game changing for me, nice to see how model learns and when to stop

[1blackbar]

but is this overfitting? the face gets totally distorted and does not resemble the original one at all those artifacts for me starts after 2000 steps not sure if this is related to this issue

you can improve these far distance full body shots by giving it like 4-5 images of phbotos of the subjects at the same distance from camera, but overall SD is not good at retaining likeness at this distance even with best learned subjects, inpaint the face with full resolution enabled in webui and move on. Aso do 640x640 not 512, youll gain extra detail without riskin doubles

[andreae293]

has any progress been made on this issue? i ran a confrontation between this repo and the xavier one same amount of steps, same regularization images, same training images, same learning rate but i get really different output and a better likeness with xavier repo

[Maximus-show]

был ли достигнут какой-либо прогресс в этой проблеме? я провел конфронтацию между этим репозиторием и xavier на одно и то же количество шагов, те же изображения регуляризации, те же обучающие изображения, та же скорость обучения, но я получаю действительно другой результат и лучшее сходство с xavier repo

Hi. can you share a link to xavier repo ?

[andreae293]

Hi. can you share a link to xavier repo ?

https://github.com/XavierXiao/Dreambooth-Stable-Diffusion

[Maximus-show]

Привет. можете ли вы поделиться ссылкой на xavier repo?

https://github.com/XavierXiao/Dreambooth-Stable-Diffusion

Thanks my friend :)