How to perform permutation statistical test?

RylanSchaeffer commented 2 years ago

Hi! I'm reading the Kriegeskorte, Mur & Bandettini paper, and Step 5 recommends using a permutation test when comparing 2 RDMs. I can't find out how to perform this permutation test using the rsatoolbox package. Can you point me in the right direction?

STEP 5: TESTING RELATEDNESS OF TWO DISSIMILARITY MATRICES BY RANDOMIZATION. We therefore suggest testing the relatedness of dissimilarity matrices by randomizing the condition labels. We choose a random permutation of the conditions, reorder rows and columns of one of the two dissimilarity matrices to be compared according to this permutation, and compute the correlation. Repeating this step many times (e.g., 10,000 times), we obtain a distribution of correlations simulating the null hypothesis that the two dissimilarity matrices are unrelated.

JasperVanDenBosch commented 2 years ago

hi @RylanSchaeffer, thanks for your question. Ideally you'd have RDMs for the underlying sample, e.g. an RDM for each participant. In this case have a look here, and keep in mind that you could consider one of the RDMs to be a "model" for the other.

For the permutation approach you mentioned, it'd be a combination of permute_rdms and compare.

RylanSchaeffer commented 2 years ago

Do I need to combine the two functions myself? Or is there a way to pass some argument to compare saying I want to perform a permutation-based statistical test?

RylanSchaeffer commented 2 years ago

Also, how do researchers nowadays evaluate RSA approaches? Do they still use permutation tests?

JasperVanDenBosch commented 2 years ago

Do I need to combine the two functions myself? Or is there a way to pass some argument to compare saying I want to perform a permutation-based statistical test?

I'll discuss with the other on whether we should build this, and get back to you asap

RylanSchaeffer commented 2 years ago

Thanks! I ask because when I compare two matrices, I want to know that (under whatever metric I select), the performed comparison uses the best statistical hypothesis test for whether the matrices are more likely independent or dependent.

nkriegeskorte commented 2 years ago

dear rylan,

the permutation test you mention is an excellent way to test for relatedness against the null hypothesis that the two representations whose RDMs you are comparing are unrelated. the test takes the dependency structure of RDMs into account and requires only the minimal assumption of exchangeability of the condition labels under the null hypothesis.

however, this is a very low bar, because different representations of the same stimuli tend to share information and this usually is reflected in some positive (though possibly low) RDM correlation. (the related notion of "distance correlation" even formally provides an indicator of mutual information between stimulus and response, which here would corresponds to model features and brain responses or two sets of model features or two sets of brain responses.)

note also that the condition-label randomization test treats subjects and experimental conditions (e.g. stimuli) as fixed effects. so you are testing for relatedness in your experimental subjects and conditions, without addressing whether the variability across subjects and conditions suggests that the correlation holds for the underlying populations of subjects and experimental conditions.

if you set test_above_0=True in the model_plot arguments, you will see a more generalizable test for positive correlation of the data RDM and the model RDM (treating subjects and/or conditions as random effects as is now standard).

moreover, theoretical progress requires that we compare representational hypotheses. the question then is whether one model better explains the data RDM than another model.

for all these reasons, the condition-label randomization test is not the most informative and high-priority thing to look at and no longer part of the standard model evaluation as visualized by the model plot function.

still it should be provided by the toolbox as an option as it can still serve useful purposes, e.g. as a sanity check when RDMs are extremely noisy, when you have too few subjects and stimuli to use bootstrapping, or when you are only interested in your sample of subjects and your sample of stimuli, rather than trying to test general hypotheses about their populations.

niko

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 6:51 AM Rylan Schaeffer @.***> wrote:

Thanks! I ask because when I compare two matrices, I want to know that (under whatever metric I select), the performed comparison uses the best statistical hypothesis test for whether the matrices are more likely independent or dependent.

— Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205092385&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=zZB_h8iklQ0lPeHq3RVuNgl5gVEqb11xVlMHelCN3QwMB9ilZXp5yu-deDaayLTl&s=q1c6RK2e37O_G17Yb66P4fGXvKVueBI7_YwVv3hZxJM&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYGNSEYALPFKRL3HXFLVXOOBVANCNFSM55RB6VHA&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=zZB_h8iklQ0lPeHq3RVuNgl5gVEqb11xVlMHelCN3QwMB9ilZXp5yu-deDaayLTl&s=Y5UmAu_3SX7g6TCMffahPhwt_fCqYli25r4ez4FlHjM&e= . You are receiving this because you are subscribed to this thread.Message ID: @.***>

RylanSchaeffer commented 2 years ago

Hi Professor Kriegeskorte,

Thanks for getting back to me! The use case I'm specifically interested in is asking which model's RDM best matches the data RDM (from a set of candidate models), or ideally, what is the rank of the models based on their RDM matches to the data RDM. What is the (sequence of) recommended analyses and best practices to answer this question?

Based on what you said above, it seems like a permutation test isn't necessarily the best approach, but what instead is?

RylanSchaeffer commented 2 years ago

I might be using terminology inconsistently with rsatoolbox. The documentation suggests that a Model predicts an RDM e.g., "The simplest type of model are fixed models, which simply predict a single RDM." By model, I mean some system that produces vector outputs for inputs that are then subsequently used to compute RDMs

RylanSchaeffer commented 2 years ago

To clarify my use case, I have $M$ candidate models and a set of $P$ stimuli $x_1, ..., x_P$, as well as neural recordings from one subject shown those stimuli. How can I rank the candidate models?

nkriegeskorte commented 2 years ago

that's exactly what the toolbox is for: inferential comparisons among multiple models.

these two papers summarize our current wisdom on this: https://nbdt.scholasticahq.com/article/27664-comparing-representational-geometries-using-whitened-unbiased-distance-matrix-similarity https://arxiv.org/abs/2112.09200

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 4:53 PM Rylan Schaeffer @.***> wrote:

To clarify my use case, I have M candidate models and a set of P stimuli {xp}{p=1}^P, as well as neural recordings from those stimuli. How can I rank the candidate models?

— Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205756216&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=8tJxV-AhjvOscFuoeMlzbonp-yHJUg9JIUeFmIH4GhU&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYBLWUJMZBBWCFQ4AYTVXQUT3ANCNFSM55RB6VHA&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=KvKRkrdueNx6Xlbq1c5VErZICWH1y6aroXPk8MNTNhk&e= . You are receiving this because you commented.Message ID: @.***>

RylanSchaeffer commented 2 years ago

Yes, that's why I'm looking at this toolbox. The problem is that I can't find a clear answer about what the best / recommended evaluation method is in the documentation, so I'm asking for directions.

Cheers, Rylan Schaeffer

On Thu, Aug 4, 2022 at 2:09 PM nkriegeskorte @.***> wrote:

that's exactly what the toolbox is for: inferential comparisons among multiple models.

these two papers summarize our current wisdom on this:

https://nbdt.scholasticahq.com/article/27664-comparing-representational-geometries-using-whitened-unbiased-distance-matrix-similarity https://arxiv.org/abs/2112.09200

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 4:53 PM Rylan Schaeffer @.***> wrote:

To clarify my use case, I have M candidate models and a set of P stimuli {xp}{p=1}^P, as well as neural recordings from those stimuli. How can I rank the candidate models?

— Reply to this email directly, view it on GitHub < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205756216&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=8tJxV-AhjvOscFuoeMlzbonp-yHJUg9JIUeFmIH4GhU&e= , or unsubscribe < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYBLWUJMZBBWCFQ4AYTVXQUT3ANCNFSM55RB6VHA&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=KvKRkrdueNx6Xlbq1c5VErZICWH1y6aroXPk8MNTNhk&e=

. You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/rsagroup/rsatoolbox/issues/249#issuecomment-1205770192, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEHLC5ZIQDC2N2CW2X653DVXQWRPANCNFSM55RB6VHA . You are receiving this because you were mentioned.Message ID: @.***>

nkriegeskorte commented 2 years ago

we're still working on the documentation and demo scripts. perhaps this demo script is a good starting point: https://github.com/rsagroup/rsatoolbox/blob/main/demos/demo_bootstrap.ipynb

heiko may have further pointers.

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 5:17 PM Rylan Schaeffer @.***> wrote:

Yes, that's why I'm looking at this toolbox. The problem is that I can't find a clear answer about what the best / recommended evaluation method is in the documentation, so I'm asking for directions.

Cheers, Rylan Schaeffer

On Thu, Aug 4, 2022 at 2:09 PM nkriegeskorte @.***> wrote:

that's exactly what the toolbox is for: inferential comparisons among multiple models.

these two papers summarize our current wisdom on this:

https://nbdt.scholasticahq.com/article/27664-comparing-representational-geometries-using-whitened-unbiased-distance-matrix-similarity https://urldefense.proofpoint.com/v2/url?u=https-3A__nbdt.scholasticahq.com_article_27664-2Dcomparing-2Drepresentational-2Dgeometries-2Dusing-2Dwhitened-2Dunbiased-2Ddistance-2Dmatrix-2Dsimilarity&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=v6fJIJIGgbxaP3HNS0HTIKtYYj_UHMlAzXnGG6cjvtQ&e= https://arxiv.org/abs/2112.09200 https://urldefense.proofpoint.com/v2/url?u=https-3A__arxiv.org_abs_2112.09200&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=6BezRmX7Noaya1dzeMR3bexWQAJyqJ98xLJHjhyU4Bs&e=

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 4:53 PM Rylan Schaeffer @.***> wrote:

To clarify my use case, I have M candidate models and a set of P stimuli {xp}{p=1}^P, as well as neural recordings from those stimuli. How can I rank the candidate models?

— Reply to this email directly, view it on GitHub <

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205756216&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=8tJxV-AhjvOscFuoeMlzbonp-yHJUg9JIUeFmIH4GhU&e=

, or unsubscribe <

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYBLWUJMZBBWCFQ4AYTVXQUT3ANCNFSM55RB6VHA&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=KvKRkrdueNx6Xlbq1c5VErZICWH1y6aroXPk8MNTNhk&e=

. You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub < https://github.com/rsagroup/rsatoolbox/issues/249#issuecomment-1205770192> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205770192-253E&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=TuYqm7k72dRfY0OIQO75dykfkAjUrBCYK2B_KuMYUZ8&e= , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACEHLC5ZIQDC2N2CW2X653DVXQWRPANCNFSM55RB6VHA> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACEHLC5ZIQDC2N2CW2X653DVXQWRPANCNFSM55RB6VHA-253E&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=M2K3EY5hE056h6yIR8SKxGII4SCMmShzoqQdrS505vk&e= . You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205778640&d=DwMFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=Sb6K-V59dT5gpnyGXTxU6HRnQedvKIC9VEY2_i-53pg&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYCY4JPIKUKPV6U7LM3VXQXN7ANCNFSM55RB6VHA&d=DwMFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=ZVY2DETT7p7XmaACLLi3HudQj0IVN3dL30afjatjJ68&e= . You are receiving this because you commented.Message ID: @.***>

RylanSchaeffer commented 2 years ago

Can one of you please clarify what a rsatoolbox.inference.result.Result's .evaluations are?

The demo doesn't really say, and the documentation just says these are "evaluations." What is the actual evaluation or test or measurement being performed?

evaluations (numpy.ndarray) – evaluations of the models over bootstrap/crossvalidation format: bootstrap_samples x models x crossval & others such that np.mean(evaluations[i,j]) is a valid evaluation for the jth model on the ith bootstrap-sample

RylanSchaeffer commented 2 years ago

Now I'm even more confused. Under the hood, eval_fixed just calls compare(rdm_pred, data, method) for each model. Where is model evaluation performed? Where is some notion of cross validation or generalization or a statistical test for significance/effect size computed?

nkriegeskorte commented 2 years ago

the model evaluations (e.g. using a correlation between data and model RDM) are performed inside the bootstrap loop. for a conceptual description of the inference, see heiko's paper linked above.

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 7:25 PM Rylan Schaeffer @.***> wrote:

Now I'm even more confused. Under the hood, eval_fixed just calls compare(rdm_pred, data, method). Where is the model evaluation being performed? Where is some notion of cross validation or generalization or a statistical test for significance/effect size?

— Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205873040&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=rAupLc46O0v-GbDwjtI2zZSHzpNZLra6SopfftFpg40ls1nXzRy_y9o3f7_qFzv9&s=UWUzMz9hbqS0bWPkVGpUgFuo30xcw1BQRDSpwSXlUDE&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYAGL2766YNEUGX3QBLVXRGQNANCNFSM55RB6VHA&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=rAupLc46O0v-GbDwjtI2zZSHzpNZLra6SopfftFpg40ls1nXzRy_y9o3f7_qFzv9&s=Oz6s6GD1Y7rIKSLvZomqEUsjbuyOMx13pMm7Z_P4Fko&e= . You are receiving this because you commented.Message ID: @.***>

RylanSchaeffer commented 1 year ago

Oh this is great - thank you!

Cheers, Rylan Schaeffer

On Thu, Aug 4, 2022 at 2:22 PM nkriegeskorte @.***> wrote:

we're still working on the documentation and demo scripts. perhaps this demo script is a good starting point: https://github.com/rsagroup/rsatoolbox/blob/main/demos/demo_bootstrap.ipynb

heiko may have further pointers.

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 5:17 PM Rylan Schaeffer @.***> wrote:

Yes, that's why I'm looking at this toolbox. The problem is that I can't find a clear answer about what the best / recommended evaluation method is in the documentation, so I'm asking for directions.

Cheers, Rylan Schaeffer

On Thu, Aug 4, 2022 at 2:09 PM nkriegeskorte @.***> wrote:

that's exactly what the toolbox is for: inferential comparisons among multiple models.

these two papers summarize our current wisdom on this:

https://nbdt.scholasticahq.com/article/27664-comparing-representational-geometries-using-whitened-unbiased-distance-matrix-similarity < https://urldefense.proofpoint.com/v2/url?u=https-3A__nbdt.scholasticahq.com_article_27664-2Dcomparing-2Drepresentational-2Dgeometries-2Dusing-2Dwhitened-2Dunbiased-2Ddistance-2Dmatrix-2Dsimilarity&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=v6fJIJIGgbxaP3HNS0HTIKtYYj_UHMlAzXnGG6cjvtQ&e=

https://arxiv.org/abs/2112.09200 < https://urldefense.proofpoint.com/v2/url?u=https-3A__arxiv.org_abs_2112.09200&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=6BezRmX7Noaya1dzeMR3bexWQAJyqJ98xLJHjhyU4Bs&e=

Nikolaus Kriegeskorte, PhD Professor of Psychology and Neuroscience Affiliated member, Department of Electrical Engineering Director of Cognitive Imaging, Zuckerman Mind Brain Behavior Institute Columbia University @.***

On Thu, Aug 4, 2022 at 4:53 PM Rylan Schaeffer @.***> wrote:

To clarify my use case, I have M candidate models and a set of P stimuli {xp}{p=1}^P, as well as neural recordings from those stimuli. How can I rank the candidate models?

— Reply to this email directly, view it on GitHub <

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205756216&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=8tJxV-AhjvOscFuoeMlzbonp-yHJUg9JIUeFmIH4GhU&e=

, or unsubscribe <

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYBLWUJMZBBWCFQ4AYTVXQUT3ANCNFSM55RB6VHA&d=DwMCaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=l9o9SjmjlM6eqLznnS-yQdcRN0bzkm7lrk_nnsEWtwN60D7OHDwPtmouRfHhha2G&s=KvKRkrdueNx6Xlbq1c5VErZICWH1y6aroXPk8MNTNhk&e=

. You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub <

https://github.com/rsagroup/rsatoolbox/issues/249#issuecomment-1205770192> < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205770192-253E&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=TuYqm7k72dRfY0OIQO75dykfkAjUrBCYK2B_KuMYUZ8&e=

,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ACEHLC5ZIQDC2N2CW2X653DVXQWRPANCNFSM55RB6VHA

< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACEHLC5ZIQDC2N2CW2X653DVXQWRPANCNFSM55RB6VHA-253E&d=DwQFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=M2K3EY5hE056h6yIR8SKxGII4SCMmShzoqQdrS505vk&e=

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rsagroup_rsatoolbox_issues_249-23issuecomment-2D1205778640&d=DwMFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=Sb6K-V59dT5gpnyGXTxU6HRnQedvKIC9VEY2_i-53pg&e= , or unsubscribe < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALWMWYCY4JPIKUKPV6U7LM3VXQXN7ANCNFSM55RB6VHA&d=DwMFaQ&c=009klHSCxuh5AI1vNQzSO0KGjl4nbi2Q0M1QLJX9BeE&r=BZiRuSoQ7OLzxH4fKsiNwbvlj0yPRTeoIkRshgziBQE&m=JfRjpNEbVsuaB6nZ-MCVEEduKGTtucC58cQq-BpJl8v85E71vp86WKGCUNDpXa3K&s=ZVY2DETT7p7XmaACLLi3HudQj0IVN3dL30afjatjJ68&e=

. You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/rsagroup/rsatoolbox/issues/249#issuecomment-1205782334, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEHLC52ENSV6OJZBEJFFPLVXQYAHANCNFSM55RB6VHA . You are receiving this because you were mentioned.Message ID: @.***>

JasperVanDenBosch commented 1 year ago

I'll close this as it seems @RylanSchaeffer 's question was resolved

rsagroup / rsatoolbox

How to perform permutation statistical test? #249