Comparative learning experiment

[denizbeser]

Now that we have different learning algorithms (Pursuit, Cross Situational and PbV), we want to compare their behavior. Specifically, we want to test the models' tolerance to noise. While all the models pass the simple object-learning tests, we expect Pursuit to be more resistant to noisy learning data, the type of language data we face in the real world.

To do this, we can test all the models with the pursuit curriculum we designed, essentially comparing their performance when they learn from the same training set. The curriculum for pursuit takes in a noise parameter, which adds noisy data points in to the training dataset (e.g. hearing "table" while there are no tables present in the scene).

Implementation details: Curriculum: To generate the curriculum, you can use make_simple_pursuit_curriculum() in pursuit_curriculum.py. You can see pursuit_object_learner_test.py line 140 for an example use case. Testing: In order to compare the models, you can first create a set of pursuit training curricula with increasing noise level. Then, iterating over the curricula, at each level of noise difficulty, you can train and test each model and report which ones succeed in learning at that level of difficulty. Parameters: The noise parameter specifies how many of the training data points will be noise data points. E.g. if num_instances is 10 and num_noise_instances is 3, there will be 3 noise learning examples and 7 regular learning examples. You can start off with a noise range of [0-5] with num_instances=10 (effectively creating up to a half-noise dataset) and then also a range with [0-10] with num_instances=20 to see the effect of training dataset size on noise tolerance. Make sure to keep other learning parameters constant in order for a fair evaluation.

[gabbard]

@denizbeser : I think for co-op students you need to be much more explciit - e.g. set up the curriculum this way, use these dimensions of variation with this particular values, look at this code for an example of how to set up something like this, etc.

[denizbeser]

@gabbard sure, I just added specifications!