Tests for different levels of noise

[aviL221]

I'm having trouble implementing tests that have the same parameters but have different levels of noise. I see several options, but they all seem imperfect: 1) Compare mock docs from tests with the same parameters but with different levels of noise (for example temporal_freq_tuning_calc.test('standard',1,1) vs. temporal_freq_tuning_calc.test('low_noise',1,1);) against the same mock expected output doc (in this example, mock.1.json) Pros: no changes needed to the generate_mock_parameters method nor to the naming of the json file of the mock expected output doc Cons: The field "individual" is an array whose dimensions depends on the number of reps specified, which can be different depending on the level of noise specified. Thus when I ran temporal_freq_tuning_calc.test('low_noise',1,1) with 10 reps, the mock output doc's "individual" field had different dimensions than the mock expected output doc, which was generated by running temporal_freq_tuning_calc.test('standard',1,1) with 5 reps. This generated an error message that was deceiving. 2) Compare mock docs from tests with the same parameters but with different levels of noise (for example temporal_freq_tuning_calc.test('standard',1,1) vs. temporal_freq_tuning_calc.test('low_noise',2,1);) against different mock expected output doc (in this example, mock.1.json for the former and mock.2.json for the latter) Pros: keeping the same naming convention for the json files Cons: The test number has to correspond to the level of noise selected, which is not intuitive. For example, temporal_freq_tuning_calc.test('low_noise',1,1); wouldn't work even though aside from the level of noise it is the same test. 3) Compare mock docs from tests with the same parameters but with different levels of noise (for example temporal_freq_tuning_calc.test('standard',1,1) vs. temporal_freq_tuning_calc.test('low_noise',1,1);) against different mock expected output doc (in this example, mock.1.standard.json for the former and mock.1.low_noise.json for the latter) Pros: no restrictions on sizes of reps, and test numbers correspond just to the set of parameters chosen and not also the level of noise. Cons: requires changing the naming convention and regenerating all existing expected docs.

I personally think option 3 is optimal for this issue, but I guess it depends on our priorities. @stevevanhooser Please let me know which of these I should implement, thanks!

[stevevanhooser]

What if we didn't compare the the number of reps in the comparison document? So that that didn't produce an error? Then could we possibly do number 1?

Alternatively, what if we used the same number of reps with each noise condition?

Thanks @aviL221

[aviL221]

That should work! I could compare the vector for data generated in each rep to the first vector in the expected doc. That way number of reps shouldn't matter. We could use the same number of reps with each noise condition, but I feel like we would benefit from having the option to have different numbers of reps depending on the noise level. But your first idea should allow for that.