EMOTE stage 1 - val loss not decreasing

[luizgh]

Hi @radekd91, thanks again for releasing the code for EMOTE!

I am training EMOTE stage 1, and I noticed that while the training loss is converging, the validation loss goes up - is this something you noticed in your runs?

[radekd91]

Hi @luizgh. yes I can confirm this behavior.

If you're using the same training/validation split, you have different individuals in training and validation set. These individuals might have vastly varying speaking styles and this can cause the vertex loss on the validation set to not be very meaningful.

If you split the dataset differently, to have seen individuals seen in both training and validation, you'd get better numbers. You can do so by setting the following split in the training script: split = "random_by_sequence_sorted_70_15_15"

[luizgh]

Ok, thanks for confirming!

I think splitting train/val to have different users is the appropriate protocol (since we want to generate motion for new identities). But yeah, I noticed this problem with other generative models too, it's quite hard to properly evaluate them. Anyway, I wanted to confirm you experience this with EMOTE to make sure there weren't other bugs on my re-run. Thanks!

[radekd91]

Yes. Evaluation of talking head avatars is really tricky and there is no easily computed number to put on it, unfortunately.

welcome to my world, the world where the only way you know if your new model is really better than the previous one is looking at generated videos and trying not to halucinate. It sucks here, you'll love it. :-D