Apply the same pre-processing to my own scanning sessions ?

[Caselles]

Hello,

First of all thanks for the great work! The progress we are seeing in the field of fMRI-to-Image is a direct consequence of your work, so big congrats.

I'm working on a research project where we do our own scanning sessions, with a similar protocol but for different tasks. The scans are done on a 3T MRI.

I wonder what is the best practice to apply the exact same pre-processing as the NSD paper but on my own scanning sessions? I don't see how this could not be done theoretically, but in practice I would like you opinion and advice on how to do it and avoid mistakes.

The goal for me would be to go from my nifti raw sessions to a dataset with pre-processed (masked, beta, etc) voxels for each event of my protocol.

I hope you'll have a bit of time to provide a bit of advice, otherwise that's completely ok and don't hesitate to close the issue.

Thanks again!

[kendrickkay]

Hello,

Regarding applying exactly the same pre-processing, that is a very high threshold to cross. The preprocessing performed in NSD is quite custom/tailored, so practically speaking it would require substantial effort to exactly replicate. That being, the code used is indeed available...

If you wanted to "more or less" replicate the pre-processing to the level of what is practically important, I think that is a much easier bar to cross. Many flavors of pre-processing out there are not that different from what NSD involved. It's just that there are details to consider, and maybe you care a lot about the details?

The NSD paper describes in full detail exactly what was done. For the most part you can think about the analysis in terms of preprocessing (e.g. head motion, spatial distortion, registration, etc.) vs. analysis (e.g. the time-series analysis that we implemented using GLMsingle). Depending on what you care about, maybe you are more concerned with the time-series analysis?

[Caselles]

Thanks for the prompt answer!

Thank you for pointing out the difficulty of exactly replicating the pre-processing. It wasn't sure that it would be that difficult and hearing that from you definitely confirms me that it is not the way to go for me.

In my research, I'm trying the following:

• start from MindEye v2, which came out recently: paper. This fMRI-to-Image model can be trained on all scanning sessions from 7 out of your 8 subjects for image reconstruction from brain data. Then, it only requires one hour of scanning data from subject 8 to learn the model how to reconstruct images from brain data, which the core contribution of the paper.
• in my case I would like to use their pre-trained model on the 7 subjects, and then fine-tune it on my data.
  • preprocessing vs analysis: in my case i'm calling pre-processing everything that is applied to the data, from its raw acquisition state to its format when inputted in the neural net (head motion, spatial distorsion, registration, betas computation, masking, etc). Sorry about that I come from a more Machine Learning background and I might not have the right vocabulary.
• every subject has a different number of voxels as input, they handle that by mapping the voxels to a shared latent space (a shared representation by the neural net).
• my guess is that if the preprocessing for my data is somewhat different from the preprocessing of the 7 subjects on which the model was pre-trained on, it'll be harder to obtain good results. That is why I'm trying to be as close in terms of preprocessing.

Hence, what do you recommend in terms of pre-processing steps? This what I had in mind :

• motion correction
• spatial distorsion
• co-registration
• compute a glm with a simple model like glover with a design matrix of the events in my sequence related to my tasks
• compute the betas for each event
• mask the betas using a mask similar to the nsdgeneral mask
• use that as input to my model

Let me know what you think!

[kendrickkay]

Thanks for the context...

Hence, what do you recommend in terms of pre-processing steps? This what I had in mind :

motion correction spatial distorsion co-registration compute a glm with a simple model like glover with a design matrix of the events in my sequence related to my tasks compute the betas for each event mask the betas using a mask similar to the nsdgeneral mask use that as input to my model

The steps you describe here sound generally fine. (Except that I don't quite know what "glover" is...)

I guess, one question to consider is the space that your data will be prepared in. Like, native volume space, or surface space, or some group space like fsaverage or MNI. Since you are trying to bridge different subjects and datasets, presumably, fsaverage is probably a good choice.

But if yyou want to lean heavily on the idea you described of mapping to some latent space, then I guess the preprocessed space won't really matter (but then of course, the details of your mapping procedure are important to think carefully about).

[Caselles]

Thank you for your answers, I'll be working this.

I'll close the issue for now, and maybe re-open it if there are further discussions to have :)

Thanks again for answering promptly, and congrats again on the great work you did.

[kendrickkay]

👍