search

Jometeorie / KnowledgeSpread

15 stars 3 forks source link

Question Regarding Evaluation Method of the Paper #2

Open yiyiyi0817 opened 1 month ago

yiyiyi0817 commented 1 month ago

Thank you very much for your work, it was very enlightening and I read them carefully from beginning to end.

I just want to ask you a small question. As far as I know, the way you calculate the accuracy rate is to ask the corresponding agent and let him answer, and make accuracy statistics according to the answer and data set.

Therefore, after benign agents are propagated counterfactual information, they store chat records in RAG, so they may retrieve counterfactual information in the process of RAG, resulting in answering counterfactual information.

Does this mean that benign agents can hardly be misled if they don't save chat logs into RAG?

Maybe my understanding of your paper is wrong. Thank you very much for pointing out my misunderstanding. And I would appreciate it if you could answer my questions. @Jometeorie @wytili

Jometeorie commented 1 month ago

Thanks for your attention!

For the first scenario, benign agents in an LLM-based multi-agent community are directly influenced by the context of the conversations, thus their cognition is directly manipulated. In the second scenario, as you mentioned, only benign agents integrated with the injected RAG system are directly affected. Once this RAG system is released on the internet (e.g., on HuggingFace), it will significantly impact other benign agents.

Additionally, agents directly influenced by the injected RAG can also help spread manipulated knowledge, thereby indirectly affecting agents not integrated with the RAG system. We leave this work for future research.

yiyiyi0817 commented 1 month ago

Thank you for your answer. Perhaps your research has revealed that for agents who design to store chat records in RAG, they need to verify whether chat records are counterfactual or toxic before deciding whether to store them.

Jometeorie commented 1 month ago

Yes, I believe this is an effective defense strategy. Agents' limited understanding of world knowledge hinders their ability to assess the authenticity of evidence, necessitating the use of external tools such as FacTool for verification.