Test NSD-Imagery retrieval performance using only similarity in raw brain space

[reesekneeland]

From Thomas Naselaris on discord:

"The brain retrieval results that @PaulScotti showed on Tuesday are really cool and seem really promising. I was thinking about why that approach might work so well, and how me might improve it. Here are some notes:

Some possible reasons why ME2 brain retrieval is working so well: -retrieval in a denoised space -contrastive denoising is especially effective -measures nearest neighbor distance in CLIP space (instead of brain activity space) -training with data from multiple subjects is really helpful

I don't have a sense of which combo of reasons are most imporant, but we could try to sort some of this out by comparing ME2 brain retrieval to other, simpler retrieval-based decoding algorithms:

-”raw” retrieval: cosine similarity between test and training patterns in noisy brain space. No denoising of any kind, no CLIP, single subject. Shouldn't work very well.

This first "raw" retrieval approach is a great work item for non-GPU enabled contributions! Retrieval opportunities appear to be rich and this is a good first step.

[reesekneeland]

This method is known to not work super well, so its main utility is as a baseline to compare against for a NeurIPS paper