search

MedARC-AI / fMRI-reconstruction-NSD

fMRI-to-image reconstruction on the NSD dataset.
MIT License
294 stars 39 forks source link

Improve low-level reconstruction pipeline #17

Open PaulScotti opened 1 year ago

PaulScotti commented 1 year ago

Can you figure out a way to beat our current metrics (for low-level pipeline) of .456 (PixCorr) and .493 (SSIM) for subject 1? Use any method you can think of to try to improve upon the current approach.

Maybe mapping to a different embedding space than Stable Diffusion's variational autoencoder? Or adopting a novel training strategy? Could even consider a ControlNet approach with multi-token textual inversion (let me know in advance if you go down that path)

One possibility: Brain-Diffuser has a low-level pipeline that maps to vdvae pretrained on imagenet-64. There is a new vdvae that came out that maps to imagenet-256. Might work better? https://github.com/ericl122333/latent-vae

mihirneal commented 1 year ago

Hey Paul, I'll be working on this issue. Currently looking towards using ControlNet in the perceptual pipeline, like the one used in CMVDM. Will keep you updated.