search

Youncy-Hu / MAGE

Make It Move: Controllable Image-to-Video Generation with Text Descriptions
Apache License 2.0
48 stars 11 forks source link

the training of VQ-VAE is ineffective #5

Open Sun-Happy-YKX opened 1 year ago

Sun-Happy-YKX commented 1 year ago

the training of VQ-VAE is ineffective. I have download "Moving MNIST datasets" by the provided code, the final folder is as follows. image But when I execute the VQ-VAE training code, the loss of test does not reduce, it keep stable at a very high level as follow. image I would like you to give me some help, thanks.

Sun-Happy-YKX commented 1 year ago

image a mistake occurs when I begin to train, add it for supplement.

Sun-Happy-YKX commented 1 year ago

image I have annoted these codes about "ckpt_path" last training, I think it is the reason of above problem. image but the "ckpt_path" is not provided, could you please give some indications.

Sun-Happy-YKX commented 1 year ago

@Youncy-Hu

Sun-Happy-YKX commented 1 year ago

image I observe that the quantization loss is expectable, but the reconstruction loss does not decrease any more. @Youncy-Hu

wxrui182 commented 1 year ago

Hello, I also encountered the same problem. The reconstruction_loss is always very high during the training process of VQ-VAE. Have you solved this problem?

wxrui182 commented 1 year ago

the training of VQ-VAE is ineffective. I have download "Moving MNIST datasets" by the provided code, the final folder is as follows. image But when I execute the VQ-VAE training code, the loss of test does not reduce, it keep stable at a very high level as follow. image I would like you to give me some help, thanks.

Hello, the problem lies in this line of code: "train_dataset = MNIST4VQVAE(args.data_root, 'train', image_transform=transform)". Just change "image_transform=transform" to "image_transform=None". The same is true for "test_dataset = MNIST4VQVAE(args.data_root, 'test', image_transform=None)". reconstruction loss

wxrui182 commented 1 year ago

image I have annoted these codes about "ckpt_path" last training, I think it is the reason of above problem. image but the "ckpt_path" is not provided, could you please give some indications.

Follow the code I modified above to train vqvae and save the model weights. Just load the weights of vqvae at this location.

wenyufeng936 commented 2 days ago

Hello, have you reproduced the results of the paper? Also, how much video memory is needed to run this experiment? Can you please let me know? thank you