search

MedARC-AI / MindEye_Imagery

MIT License
3 stars 0 forks source link

Perform ablation study on NSD-Imagery #5

Open reesekneeland opened 4 months ago

reesekneeland commented 4 months ago

MindEye2 does not scale robustly to imagery. We should spend some time performing an ablation study of the different modules and configurations to see if we can change that, and get it working effectively on imagery data.

Its performance dropoff is larger for simple stimuli than complex, so it’s weaker in representing structural content, however, it still drops off a lot for the complex stimuli as well, so giving it a stronger focus on a blurry recon should help but likely won’t be a complete solution.

reesekneeland commented 2 months ago

Number one thing to try for this is to map the ME2 architecture to versatile diffusion 257x768 clip vectors, and use VD to reconstruct. We want multi-subject pretraining with ME1 performance or greater.