Closed skywo1f closed 3 years ago
in train_vae() do a print(len(generator)) around line 160. Is it not finding all your data?
70242 ... looks about right for number of images
which if the batch size is 16, 16*4500 = 72000... so it seems like its failing at starting the second epoch?
Maybe there's an off-by-one error going on. The epoch loop might need to be for epoch in range(1, args.epochs + 1)
that seems to have fixed it, thanks!
I seem to be having an issue training the vqvae. It runs fine until about step 4300-4500 then it suddenly quits: saving complete epoch 1/2 step 4100 loss: 0.000803 - 5it/s epoch 1/2 step 4200 loss: 0.000568 - 5it/s epoch 1/2 step 4300 loss: 0.00045 - 5it/s
any ideas?