Questions about data splits and training details

[neerajwagh]

Hello -

Thank you for putting this benchmark together and releasing all the code! This is certainly an aspirational level of research and code transparency! :) I don't have any issue/bug to report with the paper or the released code.

It'd be great if the authors could comment on the following:

1. Would cropped decoding make a significant difference in training stability/dynamics compared to trialwise decoding?
2. Were the subject splits done using some form of stratification by age group?
3. What is the rationale when normalizing EEG across multiple subjects? From what I understand, the data mean remained untouched during normalization/scaling.
4. Are the models trained using the 10s-level MAE or the subject-level MAE?
5. During prediction, would there be two levels of averaging to compute the reported MAE? 1) across multiple crops to get epoch-level predictions; 2) across multiple epochs to get subject-level predictions, then compute final MAE?
6. Would training on one dataset and testing on another as additional model validation make sense? In this case, recovering unscaled predictions made on an unseen dataset based on train set age mean/stdev may give incorrect/negative ages due to age distribution differences. Is there an alternative way to normalize age targets for cross-dataset evaluations?

Thank you for your time!

--Neeraj

[agramfort]

@gemeinl can you help here?

[neerajwagh]

(Edit: I'd made an error when z-scoring the targets, which caused the high validation set loss variance. The ShallowNet on TUAB results have now been reproduced! I've removed that part from my original post. I apologize for the confusion.)