Missing weighted average function for virtual answer word

[MrZilinXiao]

Hi. Thanks for your great work.

The paper mentioned weighted average function on page 4, indicating embeddings of virtual words should be initialized with respect to the probability distribution. However, your code shows only a mean operation was performed. Is that a bug or does it just shows a negligible difference so that we could ignore it?

Moreover, I am a little confused about the probability distribution. Is it still based on prior distributions discussed in Entity Knowledge Injection in Part 4.1?

Thanks in advance for your patience.

https://github.com/zjunlp/KnowPrompt/blob/8734c20b0e6b771a747013d3399afec02365f39f/lit_models/transformer.py#L167

[njcx-ai]

Thanks for your attention. Refer to the snippet ”we adopt the weighted average function for 𝜙𝑟 to average embeddings of each words among C𝑟 to initialize these relation embeddings, which can inject the semantic knowledge of relations.“, the mean operation was performed for the initialization of relation embeddings (virtual label words). While the prior distributions discussed in Entity Knowledge Injection in Part 4.1 is mainly designed for virtual type words.

[MrZilinXiao]

Wow. Thanks for your quick response. That helps a lot! Just to be clear, so the 𝜙𝑟 you mentioned is just an average operation without any extra parameters. Got misled by the probability distribution over the candidate set C𝑟. Has your group tried assigning learnable parameters to each separate word?

[njcx-ai]

Yes, the 𝜙𝑟 mentioned is just an average operation without any extra parameters. We have not tried optimizing learnable parameters for each separate word. And this may be a direction toward the better performance of KnowPrompt.

[MrZilinXiao]

Great. Thanks for the clarification. Wish you guys success in future research careers!