What to do at inference time?

[ricshaw]

After we have trained a model with appearance embeddings, do we need to keep the embeddings for inference? Or can they be discarded? I.e. is it better to choose one of training embeddings to render a novel trajectory, or can we simply set the embedding to None and ignore the whole MLP?

[jkulhanek]

Hi, it depends on your data and goal. There is no notion of what is "better". Here are my recommendations:

• If you discard the MLPs, the result is a kind of "mean" appearance. You would lose highlights if they change in the data, but I would expect this model to score the best performance in terms of quantitative numbers.
• If you want to get best-looking video, keep the MLP and pick one training embedding (you can bake the Gaussian attributes back to the Gaussians and drop the MLP).
• If your data contain only small appearance changes (like exposure changes), it should be ok to drop the MLP after training and use the mean appearance. E.g., if there are only exposure changes, even highlights and shadows should be preserved.
• If you have large time-of-day/season changes, you should always keep the MLP and pick one/many training embeddings for rendering the novel trajectory.