Questions about the result in your paper？

[ShqWW]

When I run your original code, I get the 90% acc, which is higher then that in your paper(76%). However, there are some mistakes in your code. When I correct some of them, the result is surprisingly lower(about 56%) than that in your paper(76%). So i suggest you upload correct code of your paper that will make your result more convincing.

[tiehangd]

Hi, Shengqi, the code is an integration of EEGNet with meta learning algorithm, could you let me know which part does the mistake come from? Didn't get 90% accuracy in my runs.

[ShqWW]

OK. Please let me show the mistakes in the code first.

1. In the 'dataloader/data_preprocessing.py' there is one more indent in line 49, 50, 71, 72 data.append(data_return[0:NO_valid_trial,:,:]) label.append(class_return[0:NO_valid_trial]) So i delete the indent before the lines so as to make the code outside the inner loop.

2. In the 'dataloader/dataset_loader_BCI_IV_c.py '

there is a serious mistake, in line 81-85

train_win_x=train_win_x.astype('float32') test_win_x=test_win_x[500:,:,:,:].astype('float32') test_win_y=test_win_y[500:] val_win_x=test_win_x[:500,:,:,:].astype('float32') val_win_y=test_win_y[:500]

I change it to

train_win_x=train_win_x.astype('float32') val_win_x=test_win_x[:500,:,:,:].astype('float32') val_win_y=test_win_y[:500] test_win_x=test_win_x[500:,:,:,:].astype('float32') test_win_y=test_win_y[500:]

and i also change the '500' to '200', because the shape of test _win_x isn't large enough (about 544).

(Also there are also some useless codes which are not annotated. But it doesn't matter.)

3. In the meta_val and meta_eval, though you use 'self.model.eval()', the weight of the model is actually changed after every iteration of the test/val. But i don't know how to deal with such problem.

Totally, when i correct 1&2, the result of code is declined so much(56%-60%). So please let me know if you discovered these mistakes and if you agree with my "debug". If you make any relevant changes in your paper, please also let me know, because the idea in your paper is novel and a convincing result will help me with my research.

Thank you!

Hi, Shengqi, the code is an integration of EEGNet with meta learning algorithm, could you let me know which part does the mistake come from? Didn't get 90% accuracy in my runs.

[tiehangd]

Hi Shengqi, thank you for the comments. The first two points on data preprocessing should have been updated on this online repo in its second update earlier. Sorry for the miss. I am getting between 74% and 78% on my server, which is the result reported in the paper. This online repo should be getting the same now. The thing you can try here is to change the validation size from 200 to 400, that should do the work. Specifically, for testing on A01, the result should be around 80% on accuracy. For point 3, this validation arrangement is natured in the algorithm, and this implementation is in accordance with the mainstream implementation. I definitely agree this arrangement makes the algorithm "take advantage of" the validation set. Which is also the reason validation set need to keep a moderate size in the previous point. Actually, I would want to rename the validation set to a different name in later versions as its functionality is really somewhere in the middle of training and validation. The classification problem is modeled as a transfer learning task here, and to be rigor, we have taken the validation set into account when we report the amount of labeled target data utilized during the transfer to make a fair comparison.

[ShqWW]

Hi！ Thanks for you reply and your new code. But I still have some questions about your code.

1. In the 'main.py'

parser.add_argument('--way', type=int, default=3)

I think the default value should be changed to '4', since there are totally 4 classes in the dataset. Do you run the code with the value '4' ? When i set the value to '4', i can't get the result of your paper(i run many times but only get about 73%-75% acc).

2. it's about the point 3 for the last comment. I think you misunderstand my point according to your last reply. I didn't mean the validation arrangement but the code itself. For example, in the line 292 of 'meta.py'. logits = self.model((data_shot, label_shot, data_query)) you write this code to get the logits value. However, when you run this code the weights of the model is changed though you have written self.model.eval(). Particularly, i change the code to self.model.load_state_dict(torch.load(osp.join(self.args.save_path, 'max_acc' + '.pth'))['params']) self.model.eval() logits = self.model((data_shot, label_shot, data_query)) That is to say, reload the weight of the model in every meta_eval iteration. After i change the code, i didn't get the result of your paper(71% test acc of my modification and 80% test acc of your origin code after the same training). The same problem also occurs in the meta_val. I guess you take a wrong realization of maml eval and maml val in pytorch?

3. One more question about the pre-train result. the paper get about the 50% test acc(in the first three lines of the ) but i only get 30-40 val acc after running your code(no modification). So what's your result of pretraining?

Here's all about my questions and i'm glad if you can reply me soon. And all the results is based on the A01 subject. I really admire the ideas of your paper. If you make any relevant changes in your paper and your code, please let me know.

Thank you!

[tiehangd]

Hi Shengqi, thank you for the input! For point 1, this default value in main.py got override by the commands from pre-train.py and meta-adapt.py, which set way=4, this value should be uneffective during the execution. So the value is not always updated when we move across different tasks. For point 2, the meta eval is kept the same from the widely used repo we forked. As I said previously, we also found the algorithm takes advantage of validation set in the process. This "taking advantage" is fair here, as we are not claiming the model can perform well solely on the very minimal target data during test, but instead, we modeled our classification problem as a transfer learning task, which use a mix of both source data and target data. And to be rigorous, these validation data has been taken into account when we count the amount of target data we use. The pre-training part currently only functions to warm up the feature extractor and its prediction layer is not well taken care of. I will take a closer look into it later on. Thank you for your interest in the work!

[tiehangd]

I deleted two comments which are personal conversations here