Question regarding the prior, v2c, and hidden variables

[song-wensong]

I have a question regarding the prior, v2c, and hidden variables in the context of a certain functionality.

Questions about prior and v2c:

If prior=True and v2c=False, does this configuration essentially prioritize reconstruction, resulting in the best possible reconstruction outcomes?

Conversely, if prior=False and v2c=True, does this configuration resemble a retrieval task, aiming for optimal retrieval performance?

In the case of prior=True and v2c=True, does this setting strike a balance between reconstruction and retrieval results?

Questions about hidden variable:

What is the effect of setting hidden to True or False when aiming for optimal reconstruction quality?

If the objective is to achieve the best possible reconstruction outcomes, would configuring the parameters as follows be advisable: prior=True, v2c=False, and hidden=True?

Thank you for your assistance and clarification!

[PaulScotti]

There is no priority variable, I assume you are referring to "prior"

prior=True means diffusion prior is being trained, False means it is not v2c=True means retrieval submodule is being trained, False means it is not hidden=True means mapping to last hidden layer of CLIP, False means mapping to final layer of CLIP

[song-wensong]

There is no priority variable, I assume you are referring to "prior"

prior=True means diffusion prior is being trained, False means it is not v2c=True means retrieval submodule is being trained, False means it is not hidden=True means mapping to last hidden layer of CLIP, False means mapping to final layer of CLIP

Apologies for the confusion; indeed, I meant "prior"

I noticed in Table 4 of the paper that the reconstruction performance of Backbone + Prior surpasses that of MindEye (0.33 BiMixCo)*. If my goal is to achieve the optimal reconstruction effect, should I configure the parameters as follows: prior=True, v2c=False, and hidden=True?

[PaulScotti]

should be true true true

[song-wensong]

should be true true true

Thank you for your assistance! When v2c = True, the loss function becomes a combination of loss_nce and (prior_mult * loss_prior) instead of just loss_prior. Despite this change, why does enabling v2c=True lead to an improvement in the reconstruction results?

[PaulScotti]

We talk about this in detail in the paper

[song-wensong]

We talk about this in detail in the paper

Thank you, I will take a closer look at the paper.

[song-wensong]

@PaulScotti Hello I have solved the following problem. Thank you!

---

I want to use text and fMRI for contrastive learning. But the text embedding dimension is 77768, so I changed [`out_dim = 257 clip_size](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/Train_MindEye.py#L259) toout_dim = 77 * clip_size, but diffusion prior cannot be used. Then I set [prior](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/Train_MindEye.py#L129) to False in Train_MindEye.py. Then I wanted to use Reconstructions.py to reconstruct the image, but I found that in [utils.reconstruction](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/Reconstructions.py#L381), the code still used [diffusion_prior](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/utils.py#L506). How should I modify it? If I directly set [diffusion_priors](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/utils.py#L492) to None, I cannot get brain_clip_embeddings. Can I make [```brain_clip_embeddings0, proj_embeddings = diffusion_prior.voxel2clip(voxel.to(device).float())``` ](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/utils.py#L497) and [brain_clip_embeddings0 = brain_clip_embeddings0.view(len(voxel),-1,768) if isinstance(clip_extractor,Clipper) else brain_clip_embeddings0.view(len(voxel),-1,1024)](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/utils.py#L500C13-L500C175) run alone and set [diffusion_priors`](https://github.com/MedARC-AI/fMRI-reconstruction-NSD/blob/main/src/utils.py#L492) to None and continue running the code below?