facebookresearch / EGG

EGG: Emergence of lanGuage in Games
MIT License
286 stars 99 forks source link

test set and decouple training and language analysis #148

Closed mbaroni closed 2 years ago

mbaroni commented 3 years ago

Is your proposal related to a problem?

EGG seamlessly integrate validation, but there is no straightforward way to include a test set.

Describe the solution you'd like to have implemented

My favorite solution would be to have an easy way to save the trained agents, so that after training we can feed them arbitrary test sets.

Describe alternatives you've considered

Perhaps the simpler alternative would be to have a test option in the training, taking a further test set as option. However, it seems like a waste of resources to have to do a full re-training every time we change the test set.

robertodessi commented 3 years ago

This would allow to perform a post-hoc language analysis decoupled from the training loop #150

robertodessi commented 2 years ago

Changes in #232 allow to load a pretrained game and perform an evaluation loop on any user-provided data, thus decoupling training and testing