Hello everyone,
I am fine tuning this repo on a dataset composed of 12045 images. Unfortunately my gpu can only handle 1 image per batch, therefore I have batch_size=1 and accumulate_grad_batches=16.
At Epoch 14, the samplegs images are actually quite good, but the sample_cfg_scale3. images are quite bad.
Is this normal or am I overfitting? Do I need to train it for more epochs? The loss is around 0.19 at this point of the training.
Being afraid of overfitting, I decided to reduce the lr to 1e-5 and change accumulate_grad_batches=6.
However, even though I am at epoch 8, I can already see good samplegs images but bad samples_cfg_scale3. images.
Hello everyone, I am fine tuning this repo on a dataset composed of 12045 images. Unfortunately my gpu can only handle 1 image per batch, therefore I have batch_size=1 and accumulate_grad_batches=16. At Epoch 14, the samplegs images are actually quite good, but the sample_cfg_scale3. images are quite bad. Is this normal or am I overfitting? Do I need to train it for more epochs? The loss is around 0.19 at this point of the training.
Being afraid of overfitting, I decided to reduce the lr to 1e-5 and change accumulate_grad_batches=6. However, even though I am at epoch 8, I can already see good samplegs images but bad samples_cfg_scale3. images.
Is this normal during training?