fMRI Encoder checkpoints

[IoSonoMarco]

Hi, are the weights of the fMRI pre-trained encoder available? I would like to use the pre-trained model as it is, without further pre-training

[IoSonoMarco]

Anyone here? :)

[romain-dufly]

Please try this checkpoint (should be the same pre-trained encoder as the one used for their paper) https://onedrive.live.com/?authkey=%21AK4ow8R9U12%2DaxM&id=802FB67C5DC88F59%2112025&cid=802FB67C5DC88F59&parId=root&parQt=sharedby&o=OneUp

[IoSonoMarco]

Hey thanks for that! From this ckpt I could retrieve weights for the encoder in code/sc_mbm/mae_for_fmri.py I guess. However, are you able to point me to the fMRI Encoder class of interest here? Just if you already have experience with it. For instance, I see MAEforFMRIContrast, MAEforFMRICross and fmri_encoder classes. I assume is the last one

[romain-dufly]

You should use MAEforFMRICross, I believe. Load the state_dict with strict=False. If you only want to use the encoder then you can specify encoder_only=True at inference time.

Also, I might have misunderstood your request at first : the link I gave you is for the auto-encoder pretrained on HCP data, but not fine-tuned using cross-attention. You would still need to run phase2_finetune_cross to get the same performance as in the paper. And of course train the diffusion model.

I'm not aware of any available completely trained model, but there is the authors previous (very similar) project MindVis https://github.com/zjc062/mind-vis that contains a pretrains folder with fine-tuned models (also for diffusion models). It also contains the starting state for diffusion models when training them yourself, so you would have needed this anyway.

I hope I was clear and helpful, I'm just doing research on the same topic!

[IoSonoMarco]

I'm actually just interested in the pre-trained fMRI Encoder stand-alone. I need it to feed in fMRI data and obtain embeddings, that's it. I will use it in a totally different application.