Retrieval task codes

[AlexCo1d]

Hi, could you show me how your codes doing the retrieval task, just the function you used is okay! Thank you!

[QtacierP]

You are welcome to use the code available at this https://github.com/QtacierP/PRIOR/blob/086bf2e0d379d7dacc47a6ad738ffabc1ad6e25f/codes/prior/downstream/zero_classification_model.py#L159

However, before utilizing it, please replace the prompt as indicated here https://github.com/QtacierP/PRIOR/blob/086bf2e0d379d7dacc47a6ad738ffabc1ad6e25f/codes/prior/downstream/zero_classification_model.py#L76

Specifically, you can modify the prompts in the dataset by using the reports from MIMIC-CXR, which can be found at https://github.com/QtacierP/PRIOR/blob/086bf2e0d379d7dacc47a6ad738ffabc1ad6e25f/codes/prior/data/zero_shot_classification/chexpert_zls.py#L58

For zero-shot classification, prompts are generated simply. However, for retrieval tasks, the primary adjustment is to use the reports as the prompts. The remaining steps are similar to zero-shot classification.

I hope you find this information helpful.

[AlexCo1d]

https://github.com/QtacierP/PRIOR/blob/086bf2e0d379d7dacc47a6ad738ffabc1ad6e25f/codes/prior/downstream/zero_classification_model.py#L183 Thank you for your reply! I noticed that this part you comment it [1,1], so i am confused that how I can process and generate a [B,B] similarity matrix based on logits, do you still remember the specfied shape of total_local_logits? Or is it already a [B,B] matrix before softmax?

[QtacierP]

You can remove the mean() here https://github.com/QtacierP/PRIOR/blob/086bf2e0d379d7dacc47a6ad738ffabc1ad6e25f/codes/prior/downstream/zero_classification_model.py#L154 text_to_local_image_atten and image_to_local_text_atten are both [B, B] similarity matrixs :)

[AlexCo1d]

Thank you a lot!