search

Alljoined / EEG_Image_decode

Using vision-language models to decode natural image perception from non-invasive brain recordings.
MIT License
2 stars 1 forks source link

Debug Alignment training step #1

Open jonxuxu opened 5 months ago

jonxuxu commented 5 months ago

We are working on the script in https://github.com/Alljoined/EEG_Image_decode/blob/newds/Retrieval/ATM_S_insubject_retrieval.py.

It works with our dataloader, but the training does not seem to be improving test accuracy:

image

vX accuracy means X-way top-1 accuracy. So the blue performance is basically chance. For some reason we got good results on one particular run, but it cannot be replicated.

Some directions: confirm the projection / transformer architecture dimensions do a sanity check on the EEG, see if there is similar response curve or ERP on responses for the same image

guneetsk99 commented 5 months ago

@jonxuxu I have picked this up already. Sorry I could not respond earlier