robintibor
commented
5 years ago First, just to make sure, you are now working on BCI Comp IV 2b not 2a, right?
1) Stuff about shallow always being same is very strange to me, really should not be like that 2) Stop criterion yeah hm it is a bit overcustomized at the moment probably. We have mostly switched to a single-phase training with cosine annealing nowaways.
About kappa computation: I don't know how you compute kappa, but it is not simply kappa over trials(!) for this competition. My understanding of how kappa was computed (independently for each subject) for this competition is: 1) From per-timestep predictions for each trial (for entire 4 sec) compute kappa independently over trials for each timestep(!). So you will have a kappa"timecourse": kappa value for first timestep, for second timestep etc. 2) Use the maximum (!) kappa value from this timecourse
I gathered that this is the way it was done from:
1) http://bbci.de/competition/iv/desc_2b.pdf See p.5, evaluation (same also for dataset 2a!). Although it still remains a bit unclear here, what it means with "with the largest kappa value" 2) https://www.researchgate.net/profile/Cuntai_Guan/publication/40450440_Multi-class_Filter_Bank_Common_Spatial_Pattern_for_Four-Class_Motor_Imagery_BCI/links/561cd9b908ae6d17308d3a9d/Multi-class-Filter-Bank-Common-Spatial-Pattern-for-Four-Class-Motor-Imagery-BCI.pdf (competition winners). See p. 573, figure 3. Here it is quite clear in my view. This is on 2a, but as written above 2a/2b follow same evaluation guidelines. 3) BioSig source code. As far as I recollect I checked my understanding is correct by looking in the source code directly where the evaluation code was still present. Feel free to recheck.
erap129
Hello, according to table VI (Decoding results for additional datasets) in your paper, The deep ConvNet should reach a kappa metric of 0.598 and the shallow ConvNet should reach 0.899. I tried replicating this result, both with my implementation and by replicating your example file "bci_iv_2a.py" and "bci_iv_2a_cropped.py". To do so I created my own Monitor class which measures the validation kappa value, and similar to your NoDecrease class I created a NoIncrease one (all this is just to use the validation kappa for early stopping). Using the deep architecture with cropped training I get an average test kappa of 0.566, and with the shallow architecture I get 0.545 (nowhere near the reported 0.899) I also found I have to change the function 'setup_after_stop_training' in experiment.py. There is a reassignment there to self.stop_criterion, and I found that if I don't change it the training will run for 1-3 epochs in the second phase and stop, giving me a test kappa of 0. I made the self.stop_criterion same as the one defined in the main example file, only with a higher max_epochs and max_increase_epochs count. On top of that - the results for the shallow network are suspicious, all subjects except subject 1 receive the exact same maximum kappa. For the deep network, it is more diverse but still, values tend to repeat.
Am I missing anything? I would like to replicate the results for this dataset but so far haven't succeeded.
Additionally - Are the results reported the maximum value of the 'epochs_df' variable in the respective column? Or is the last value taken? From my runs usually, the last value isn't necessarily the highest measured.
Thank you very much, Elad