search

tiehangd / MUPS-EEG

Efficient Transfer Learning with Meta Update for Cross Subject EEG Classification
38 stars 15 forks source link

About a serious problem in your code. #3

Closed ShqWW closed 4 years ago

ShqWW commented 4 years ago

After discussing with you in another issue, I am almost sure that there is a serious problem in your code. Let me show you the problem as follows.

The problem is in the line 199&292 in 'meta.py'

  1. logits = self.model((data_shot, label_shot, data_query))

  2. logits = self.model((data_shot, label_shot, data_query))

These two lines should be changed as follows:

torch.save(self.model.state_dict(), './tem.pth') logits = self.model((data_shot, label_shot, data_query)) self.model.load_state_dict(torch.load('./tem.pth')) self.model.eval()

torch.save(self.model.state_dict(), './tem.pth') logits = self.model((data_shot, label_shot, data_query)) self.model.load_state_dict(torch.load('./tem.pth')) self.model.eval()

And after changing the code, I find the result is about 42% on both test and validation dataset(Sub A01). So you can try my modification and see if you can get the same result. And i think the repo 'meta-transfer-learning' also have the same problem (pytorch version), though it's forked by so many people.

I'm very sorry to pick up such a problem because i was inspired by the idea in your paper before. If you think my modification is correct but useless(no distinguish between the origin code and the modified code), please explain why the actual result becomes worse. If you think my modification is incorrect, please explain why it's incorrect.

Thank you!

tiehangd commented 4 years ago

Hi Shengqi, as I said in your previous post, the implementation of meta eval is kept the same with other widely used repos. This "taking advantage" is fair here, as we are not claiming the model can perform well solely on the very minimal target data during test, but instead, we modeled our classification problem as a transfer learning task, which use a mix of both source data and target data. And to be rigorous, these validation data has been taken into account when we count the amount of target data we use. If we make the suggested modification, then the only target data we use will be the very minimal number of shots during test, which is not our transfer setting. For the transfer learning scenario introduced here, you can think of it as an meta integration of inter subject training and intra subject training.