In this PR we add microsaccades to Brain-Score core.
Conceptual Problem: We need variance in model responses to static stimuli.
Conceptual Solution: Microsaccades are a prevalent phenomenon in vision and visual experiments. They are involuntary movements of the eye at very small scales. Microsaccades are not (and often can not due to their small extent) be controlled for in experiments, and are thus implicitly baked into the variance in primate responses. As such, microsaccades are a phenomenon that is either explicitly or implicitly believed to be unimportant for behavioral/neural outcomes, and thus a good candidate for getting variance in responses to static stimuli. (e.g. Rolfs 2009, Vis. Res.; Haddad & Steinmann 1973, Vis. Res.)
Technical problem: The current BrainScore interface does not support microsaccades.
Technical solution: the ActivationsExtractorHelper will be changed. The from_paths method now takes a path, loads the image from the path, does pixel position jittering to the image, and saves the jittered image into a temporary new path. The new path is passed to the downstream functions to get the model activations.
To accomplish this, the from_paths method will now receive a dictionary of tags. This dictionary of tags comes from a call to the look_at method, where number_of_trials is replaced with a new dictionary. This new dictionary then traverses the following path:
In this PR we add microsaccades to Brain-Score core.
Conceptual Problem: We need variance in model responses to static stimuli.
Conceptual Solution: Microsaccades are a prevalent phenomenon in vision and visual experiments. They are involuntary movements of the eye at very small scales. Microsaccades are not (and often can not due to their small extent) be controlled for in experiments, and are thus implicitly baked into the variance in primate responses. As such, microsaccades are a phenomenon that is either explicitly or implicitly believed to be unimportant for behavioral/neural outcomes, and thus a good candidate for getting variance in responses to static stimuli. (e.g. Rolfs 2009, Vis. Res.; Haddad & Steinmann 1973, Vis. Res.)
Technical problem: The current BrainScore interface does not support microsaccades.
Technical solution: the ActivationsExtractorHelper will be changed. The from_paths method now takes a path, loads the image from the path, does pixel position jittering to the image, and saves the jittered image into a temporary new path. The new path is passed to the downstream functions to get the model activations.
To accomplish this, the from_paths method will now receive a dictionary of tags. This dictionary of tags comes from a call to the look_at method, where number_of_trials is replaced with a new dictionary. This new dictionary then traverses the following path:
TODO list:
Other Brain-Score benchmark PRs depend on this PR (Malania2007).
Huge thanks to Hannes Mehrer (@hanme) who implemented microsaccades, which I've only adapted for the generalized case here.