search

dongyangli-del / EEG_Image_decode

Using vision-language models to decode natural image perception from non-invasive brain recordings.
MIT License
78 stars 15 forks source link

Questions about ATM_S_insubject_retrieval.py file #18

Closed xiuwenwu closed 2 months ago

xiuwenwu commented 2 months ago

I noticed that when training the ATM_S insubject retrieval task, there are two models involved in this parameter update:

  1. eeg_model; 2. img_model. From what I understand, the img_model is clip model for img features extraction. Will updating the clip network parameters during the training process improve the performance of the model a lot? pngs
dongyangli-del commented 2 months ago

Hi, @xiuwenwu, the CLIP image encoder is called in eegdatasets_leaveone.py. The definition of img_modelshown in your image is only used for testing purposes. In the actual process, img is not projected because training the projector would cause the original knowledge of CLIP to be forgotten.

Therefore, just ignore this function img_model.

xiuwenwu commented 2 months ago

Thank you for your response!!!