关于可视化和数据

[Fengww2014]

你好，按照我的理解，这个数据的划分是不是测试和训练是不存在重叠的类别，测试是200个训练中没有的类别，那么问题来了，怎么保证能够生成未知的类别的呢？我也看了一下你发到google drive的图像，很明显语义并不是特别的好啊，这个测试训练的划分本质上就不太合理啊，难道不是最理想的状态是我连续10天采集a的脑电信号，头九天是训练，第十天测试吗？

[dongyangli-del]

Hi, @Fengww2014, in the field of neural engineering, we usually specifically refine a single problem across subjects, sessions, and devices to achieve better performance. However, in the experimental setting of the THING-EEG dataset, it is clear that cross-sessions decoding is not the main purpose.

In our paper, we only provide a method for solving EEG decoding and visual reconstruction. Due to the scarcity of datasets in this field, there is currently no large amount of data for us to evaluate the performance of other dimensions, such as across sessions. Why does our method have zero-shot capability? I suggest you read the CLIP paper.

Related work:

1. A large and rich EEG dataset for modeling human visual object recognition.

2. DECODING NATURAL IMAGES FROM EEG FOR OBJECT RECOGNITION.

3. Learning Transferable Visual Models from Natural Language Supervision.

[Fengww2014]

我理解你说的意思，但是我想说的是测试和训练存在很强的语义gap，我们想象一个极端的案例训练只有苹果类别，测试让你去测试香蕉，我觉得这个很奇怪啊。纵使训练集有橘子、饼干、只要没有香蕉，我觉得都很难重构出香蕉啊。分布都不一样了啊。clip的zore shot的能力是因为它具备image 和text的预训练，你用clip去测试一下小语种或者一种未发现的物种很明显他不具备这种泛化性啊。

[Fengww2014]

我对你们的工作非常认可，很棒的工作，当时对于这个任务本身我感到一些疑问

[dongyangli-del]

Hi, @Fengww2014, in the CLIP article, the definition of zero-shot is that the test set allows images that have never appeared in the training set. But there is a condition here, that is, the text description of the test set samples is required to calculate the similarity.

In our work, we use contrastive learning loss to achieve zero-shot retrieval and classification tasks, and with the help of reconstruction loss, as well as high level and low level control, we can achieve image reconstruction of similar objects using only EEG. Of course, it is impossible to reconstruct objects that are exactly the same as the original image, because the categories in the test set have not been seen in the training set.

[Fengww2014]

对的，作为检索和分类任务你利用对比学习得到的encoder就可以完成了，但是作为重构任务，势必无法重构出来的。things-eeg这个数据集的划分就不太适合作为重构任务，适合作为一个理解的任务。

[Fengww2014]

这也就是我说我看这个文章最疑惑的地方，你们的做法方法都挺好的，不知道你们实验室有没有搞一个数据集的计划

[dongyangli-del]

Coincidentally, we were inspired by the work presented in the paper "Brain decoding: toward real-time reconstruction of visual perception". The authors utilized the THINGS-MEG dataset to retrieve and reconstruct new classes that had never been seen in the test set. This approach was not only innovative but also highly inspiring to us.

Influenced by recent work, we hope to achieve similar results on the THING-EEG dataset, which would significantly expand the practical applications of brain-computer interfaces. To this end, we have tested the effects of both THING-EEG and THINGS-MEG in our paper. Our goal is to develop a unified framework that makes decoding and reconstruction of brain-computer interfaces more cost-effective and practical.

This unified approach aims to bridge the gap between theoretical research and real-world applications, making brain-computer interfaces more accessible and beneficial to a broader audience.