Intermediate Checkpoints are not being saved

[mandal-rahul]

Im running for 4000steps with saving model every 500steps starting from 1500. But still it doesn’t save the intermediate steps and just at the end of 4000steps it dumps the final checkpoint in gdrive

[TheLastBen]

Did you check the box "Save_Checkpoint_Every_n_Steps" ?

[mandal-rahul]

Yes I did, still it wont save the ckpt files. Also im using the old method

[TheLastBen]

I'll check it out

[mauzus]

Yes, I just tried the old method too, and it didn't save any intermediate checkpoints.

[TheLastBen]

Did you test it with the new method ?

[mandal-rahul]

The new method doest work perfectly, but the old method doesnt save intermediate checkpoints

[TheLastBen]

I'll fix the issue with the old method

I'll walk you through the new method, just tell me how many instance images you have and what subject you're training

[mandal-rahul]

Thanks, seems like the old method is working fine now. I have 130 instance images, and the subject is man. In the new approach I have trained for 9k steps with saving ckpt every 500 starting from 1000. I did evaltuation of each ckpt (starting from 1000 steps till 9k steps) but for me the old approach gives me best result.

[TheLastBen]

Try this :

• Pick the best 15 pictures from your dataset, rename all of them to "pbftefbdg", so you'll get : pbftefbdg (1) .... pbftefbdg (15),
• Run the new method cells, and upload the 15 images, make the images contain closeups.
• Set the steps to 2000 (should take 30-40 minutes), keep the box fp16 checked.

When done, try this exact prompt :

(pbftefbdg), award winning photo by Patrick Demarchelier , 20 megapixels, 32k definition, fashion photography, ultra detailed, precise, elegant

Negative prompt: ((((ugly)))), (((duplicate))), ((morbid)), ((mutilated)), [out of frame], extra fingers, mutated hands, ((poorly drawn hands)), ((poorly drawn face)), (((mutation))), (((deformed))), ((ugly)), blurry, ((bad anatomy)), (((bad proportions))), ((extra limbs)), cloned face, (((disfigured))), out of frame, ugly, extra limbs, (bad anatomy), gross proportions, (malformed limbs), ((missing arms)), ((missing legs)), (((extra arms))), (((extra legs))), mutated hands, (fused fingers), (too many fingers), (((long neck)))

Steps: 45, Sampler: DPM2 a Karras, CFG scale: 8.5, Seed: 2871323065, Size: 512x704, Model hash: ef85023d, Denoising strength: 0.7, First pass size: 0x0

Let me know the result

[mandal-rahul]

sure thing, will give it a try tomorrow

[mauzus]

It's still broken, unfortunately. Sometimes it works, sometimes it doesn't for some reason... It's so painful to waste a lot of time and only get an overtrained model. 😭

I'll have to learn the new method and see if I have better luck. 😰

[TheLastBen]

The new method doesn't need to be learned, it's only 2 steps