search

muzairkhattak / multimodal-prompt-learning

[CVPR 2023] Official repository of paper titled "MaPLe: Multi-modal Prompt Learning".
https://muzairkhattak.github.io/multimodal-prompt-learning/
MIT License
635 stars 48 forks source link

Prompts layer #72

Closed nwlxcdh255 closed 2 months ago

nwlxcdh255 commented 3 months ago

Hi, your paper says there are prompts in the front L layer But I see your code should not be layer 1 is there no prompt Which means there's an L-1 prompts? Looking forward to your reply

if not self.first_layer: if len(compound_prompts_deeper) > 0:

This means that deeper compound prompts are turned on

Here it behaves differently for text and visual side

Forward function is same for both

if not self.text_layer:

First check if the ith layer needs compound prompts or not

if not (counter > len(compound_prompts_deeper) - 1):

muzairkhattak commented 3 months ago

Hi @nwlxcdh255,

Thank you for showing interest in MaPLe!

Regarding your question, kindly note that we are using prompts in the first L layers.

Actually, in the MaPLe trainer code, we have already used prompts for the first layer (as shown in these lines at maple.py at this line).

Therefore, we do not use additional prompts in the first layer when we are inside model.py (that is why we have the if not self.first_layer:).

So overall we are using learnable prompts in the first L layers.

I hope that is clear now. Let us know if there are any questions.

Thank you and kind regards!

dana-niu commented 2 months ago

Thank you very much for your reply.