Questions about how to create learnable latent code at test time

[friku]

Thanks for the great research! During training, the latent code is learned for each frame, but during testing, there is no latent code learned. As far as I can see, latent codes are all set to 0 or on commented out line, there are signs that the average of the latent codes during training was tried. I would appreciate it if you could tell me how you create the latent code at test time.

[gafniguy]

Hey, as the latent codes are learned, they cannot be created in test time; the idea is to use the space of latent codes learned in training. Practically, we just randomly pick a frame from training and use its single latent code to inference the entire test sequence.

The latent codes are saved within the model checkpoints, and the eval script loads that and just picks one. The avg and zero ones are just experimental.