One of the limitations of brain retrieval is that because ME2 maps to a clip embedding (either SDXL unCLIP or ViT/L last layer), we are potentially not mapping to an embedding that would be useful for doing retrieval when we only care about structural information, as is the case in the simple stimuli of NSD-Imagery.
This task is a little more open-ended, but the goal is to find a pretrained embedding space that captures low level structure of images, and to train ME2 to map brain activity patterns to this space by adding a new loss function, so that we can perform brain retrieval that is more sensitive to low level features, and will hopefully improve simple stimuli retrieval results.
One of the limitations of brain retrieval is that because ME2 maps to a clip embedding (either SDXL unCLIP or ViT/L last layer), we are potentially not mapping to an embedding that would be useful for doing retrieval when we only care about structural information, as is the case in the simple stimuli of NSD-Imagery.
This task is a little more open-ended, but the goal is to find a pretrained embedding space that captures low level structure of images, and to train ME2 to map brain activity patterns to this space by adding a new loss function, so that we can perform brain retrieval that is more sensitive to low level features, and will hopefully improve simple stimuli retrieval results.