Overfitting when training lstm model!

[sasukepn1999]

Hi @cspampin, thanks for your great work firstly. Currently, I've trained the model (lstm) in your dataset. But after training for 100 epochs, the model was overfitting, TrL was decreasing while VL and TeL were increasing, and TeA at max VA = 0.1457 at epoch 28. I wonder how can I fix the problem to achieve good performance (as result in the paper).

[dara1400]

Hi I also have the same problem and same result.

[cspampin]

Hi, the results in the paper are affected by slow frequency in EEG data. When filtering properly the data, the results for lstm are about 15-20% as you obtained.

[dara1400]

Thanks for your response. For the second part of the paper 2000 images are needed. The ImageNet dataset is so large and I can’t download it to separate the images. Did you uploaded these images in any repository?

[cspampin]

We cannot release ImageNet data for licence issues. What we can do is to share the class names. Sorry.

[dara1400]

Thanks for your response. Yo reported above 80 % accuracy for test and validation sets. How is that possible? You said here he results for LSTM are about 15-20%

[cspampin]

If you run the code on the raw data you will get 80% of accuracy, if you filter the data from 1Hz to 45Hz you obtain the 20% of accuracy.

[dara1400]

Thanks a lot I will test it.

[prajwalsingh]

@cspampin So, is the accuracy reported in the paper for CVPR and TPAMI using raw eeg signals?

[YutLan]

@cspampin Is the file named "eeg_signals_raw_with_mean_std.pth" the raw data you mentioned? I still can not reproduce the results. @dara1400 what's your process? Thanks!

[prajwalsingh]

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper.

It will be helpful if you crosscheck this from your end.

[YutLan]

@prajwalsingh Thank you very much! I have downloaded the dataset. Does the raw data mean the file "eeg_signals_raw_with_mean_std.pth"?

[prajwalsingh]

@YutLan , Yes.

[YutLan]

@prajwalsingh Thank you! It will be helpfu! Maybe my implementation misused the early stop. I will crosscheck this from my end!

[YutLan]

By the way, Nice to meet you! @prajwalsingh I've actually read your team's eeg2image paper before. What a coincidence!

[prajwalsingh]

Thank you 😊, good to know that.

[YutLan]

I have searched the learning rate, seed with the raw data. The best results on the valid and test dataset are about 40%. Would you so kind to tell me your experimental details? @prajwalsingh I downloaded data from here. Thank you!

[prajwalsingh]

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/19210048_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

[YutLan]

Thank you! I will reply as soon as possible!

[YutLan]

@prajwalsingh TrL=0.7866, TrA=0.7243, VL=1.6355, VA=0.5029, TeL=1.6905, TeA=0.5020, TeA at max VA = 0.5312 at epoch 2045, it seems that it not work and maybe I should increase the epochs. Would you be so kind as to share the data link you used?

[prajwalsingh]

@YutLan I have used the same dataset. I downloaded the code from this repository only. Try changing lstm depth from 1 to 2 or 4. You may get acc around 94% on test data.

[YutLan]

@prajwalsingh Thanks, It works!

[NorbertZheng]

I just visualize some EEG signals belong to the same class, using

dataset = torch.load("eeg_signals_raw_with_mean_std.pth")
plt.imshow((dataset["dataset"][0]["eeg"].numpy() - dataset["means"].numpy()) / dataset["stddevs"].numpy())
plt.show()

subj 4 & label 10

>>> print([data_idx for data_idx in range(len(data)) if data[data_idx]["subject"] == 4 and data[data_idx]["label"] == 10])
[0, 3, 31, 215, 216, 480, 499, 517, 580, 598, 599, 618, 646, 682, 719, 860, 886, 919, 940, 953, 958, 1030, 1059, 1107, 1114, 1130, 1211, 1243, 1256, 1336, 1342, 1367, 1522, 1549, 1594, 1653, 1693, 1702, 1706, 1720, 1728, 1751, 1798, 1817, 1829, 1913, 1938, 1957, 1959, 1960]

subj 4 & label 15

>>> print([data_idx for data_idx in range(len(data)) if data[data_idx]["subject"] == 4 and data[data_idx]["label"] == 15])
[56, 117, 179, 207, 234, 244, 276, 290, 454, 466, 487, 513, 534, 548, 584, 665, 795, 818, 861, 878, 907, 1006, 1048, 1055, 1072, 1101, 1110, 1117, 1266, 1307, 1309, 1365, 1391, 1409, 1422, 1441, 1495, 1544, 1640, 1663, 1668, 1679, 1691, 1705, 1773, 1790, 1889, 1898, 1966, 1977]

[NorbertZheng]

Is that normal?

[NorbertZheng]

I also visualize some EEG signals belong to the same class of eeg_5_95_std.pth dataset, using

dataset = torch.load("eeg_5_95_std.pth")
plt.imshow((dataset["dataset"][0]["eeg"].numpy() - dataset["means"].numpy()) / dataset["stddevs"].numpy())
plt.show()

subj 4 & label 10

>>> print([data_idx for data_idx in range(len(data)) if data[data_idx]["subject"] == 4 and data[data_idx]["label"] == 10])
[0, 3, 31, 215, 216, 480, 499, 517, 580, 598, 599, 618, 646, 682, 719, 860, 886, 919, 940, 953, 958, 1030, 1059, 1107, 1114, 1130, 1211, 1243, 1256, 1336, 1342, 1367, 1522, 1549, 1594, 1653, 1693, 1702, 1706, 1720, 1728, 1751, 1798, 1817, 1829, 1913, 1938, 1957, 1959, 1960]

subj 4 & label 15

>>> print([data_idx for data_idx in range(len(data)) if data[data_idx]["subject"] == 4 and data[data_idx]["label"] == 15])
[56, 117, 179, 207, 234, 244, 276, 290, 454, 466, 487, 513, 534, 548, 584, 665, 795, 818, 861, 878, 907, 1006, 1048, 1055, 1072, 1101, 1110, 1117, 1266, 1307, 1309, 1365, 1391, 1409, 1422, 1441, 1495, 1544, 1640, 1663, 1668, 1679, 1691, 1705, 1773, 1790, 1889, 1898, 1966, 1977]

[NorbertZheng]

• Do you have the preprocessing script to get eeg_5_95_std.pth dataset from eeg_signals_raw_with_mean_std.pth dataset?

Many thanks!!! :)

[Zoe-Wan]

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper.

It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

[prajwalsingh]

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper. It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples.

You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

[Zoe-Wan]

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper. It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples.

You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

Oh! Thank you for your guidance. I realized I was reading an older version of the paper. Does the new version propose corrections to the original experiments? I will read it to understand. I will also review your work. If there are areas I don't understand, I hope to seek your guidance. :) Thanks again!

[Zoe-Wan]

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper. It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples.

You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

I find something interesting 🤣 . In another TPAMI paper The Perils and Pitfalls of Block Design for EEG Classification Experiments, it said that "We take our correspondence with the authors to imply that no filtering was applied during acquisition, no filtering was applied prior to production of either the Python or Matlab format released data". And I have found some more papers which think this dataset has some flaws. Maybe I should change the dataset 😢

[prajwalsingh]

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper. It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples. You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

I find something interesting 🤣 . In another TPAMI paper The Perils and Pitfalls of Block Design for EEG Classification Experiments, it said that "We take our correspondence with the authors to imply that no filtering was applied during acquisition, no filtering was applied prior to production of either the Python or Matlab format released data". And I have found some more papers which think this dataset has some flaws. Maybe I should change the dataset 😢

Interesting, thanks for sharing the details. You can check the ThoughtViz paper and one more object EEG dataset is there. I think there are few papers in CVPR21 that proposed the EEG dataset.

[bettermanc]

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

[prajwalsingh]

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

Hi, I have updated the [link].

[bettermanc]

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

Hi, I have updated the [link].

Wow, thanks for your quick reply, I will try this code!

[bettermanc]

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

Hi, I have updated the [link].

Hello, could you please share your opinion about dataset splitting? I noticed that the paper mentioned, "Our dataset was split into training, validation, and test sets, with respective fractions 80% (1600 images), 10% (200 images), 10% (200 images)." Does this mean that the final size of the training set is 16006=9600, the validation set is 2006=1200, and the test set is 200*6=1200? However, in the file block_splits_by_image_single.pth, the lengths of the three are 669:167:164, and in block_splits_by_image_all.pth, the lengths are 7970:1998:1997. These are not in an 8:1:1 ratio. What is the reason for this? Many thanks!

[woldier]

@cspampin Is the file named "eeg_signals_raw_with_mean_std.pth" the raw data you mentioned? I still can not reproduce the results. @dara1400 what's your process? Thanks!

Hi YutLan, I read your "Seeing through the Brain" paper before. I'm trying to download the dataset from OneDrive now, but I'm finding that the download is unusually slow in mainland China, and I've been unable to get it to download successfully. Could you please share a download link with me privately? Thank you very much!