search

brainhack-school2020 / maellef_NeuroDash

Dashboard to help the visualisation of BIDS Datasets
MIT License
0 stars 0 forks source link

models type and tasks #1

Open simonsoubeyrand opened 4 years ago

simonsoubeyrand commented 4 years ago

Looks like a cool project!

Regarding your goal to model brain data with AI, did you plan on specific models already available (e.g. vgg16, resnet) or custom ones?

Also, how did you plan to model these processes; through prediction of BOLD signal?

maelleF commented 4 years ago

Thanks!

For the model, we used a technique called Transfer Learning: We used a network that has already be trained on audio (SoudNet in that case, https://projects.csail.mit.edu/soundnet/), but we took the output from one of the network internal layer, and give it to a custom network (a really simple one for now, some convolutional layer and a relu!). We plan on testing some more complex model, maybe with some rnn.

For the prediction, we don't use the BOLD signal directly, as it would be too much for the network to learn. Instead, We use à parcellation of the signal, so for each time point we only have 210 values for the selected 210 ROI

Again, thanks for the question!

simonsoubeyrand commented 4 years ago

My pleasure. And indeed parcellation looks like a better way to go for this type of data.

I think transfer learning might be the way to go for the last step of my project too. @PeerHerholz , what do you think? I could fine-tune AlexNet to predict, say, the RDMs (49x49 matrix) of the participants? This would be after simply trying to feed it with our image set and link with these RDMs to check the (brain x dnn) correspondance in hierarchical processing.

Thanks Maelle, Cheers.

PeerHerholz commented 4 years ago

Great conversation and points @maelleF and @simonsoubeyrand!

You're definitely asking the though questions here.... I think this is hard to speak about in a GitHub issue as it will get very long and extensive. How about we organize a call during next week's hacking time to get the AI pros involved as well?

One thing that needs to be addressed is "representation" itself and describing vs. explaining, etc. . Of course you can compare brain and DNN RDMs in a layer-ROI fashion, but using that information to fine-tune a given network is not super straight forward as you won't have information on which features or which (most likely non-linear) combination of them (if at all) "guides" a certain representation. You could think about incorporating feature based RDMs to check this further...

@emdupre this is your stuff! Any pointers? Also tagging @pbellec, @k-jerbi, @jbpoline.

maelleF commented 4 years ago

Seems like a really great discussion indeed ! I'm for a call so we can discuss further.

PeerHerholz commented 4 years ago

@maelleF should we maybe schedule a call on Thursday/Friday to discuss this with whoever is interested?

simonsoubeyrand commented 4 years ago

Yes! I can do both days.