Resuming training produces identical images/no progress

[kisenera]

After resuming from a stem network, the fakes will not change. The training continues and snapshots get produced, but they are visually identical to previous ones. (e.g., every snapshot will mimic fakes_init)

[xl-sr]

Can you post the command your using to continue from the stem? It sounds like you're not actually adding any new layers.

[kisenera]

Oh, do you need to add new layers if you just want to continue working on the stem and aren't upscaling? I used this

train.py --syn_layers 7 --cfg stylegan3-t --gpus 1 --snap 1 --metrics none --workers 4 --mirror 0 --batch 2020 --batch-gpu 20 --data scenes16.zip --outdir traning-runs --resume N:\Downloads3\stylegan-xl\training-runs\00000-stylegan3-t-scenes16-gpus1-batch2040\network-snapshot.pkl

[xl-sr]

so you are just continuing training? In this case, you can just run the same command without --resume, it will automatically continue training if you do not change the input arguments.

[nom57]

@kisenera your batch size is way too high and can be detrimental on diversity. dont go above batch=128.