search

bbaaii / DreamDiffusion

Implementation of “DreamDiffusion: Generating High-Quality Images from Brain EEG Signals”
MIT License
429 stars 49 forks source link

confusion about CLIP #14

Open microcourier opened 8 months ago

microcourier commented 8 months ago

Thanks for your great paper. I have a confusion about the CLIP part and wonder if you can clarify it.

In the paper, part 3.3 shows that the EEG features are Aligned by CLIP, but i don't realize where the code is.

i will appreciate it if you could clarify it. Thank a lot.

alinvdu commented 7 months ago

get_clip_loss function from cond_stage_model is running a cosine similarity between target embeddings and image embeddings. Latent embeddings are returned by get_learned_conditioning.

Hope it helps.