Open Cooperx521 opened 6 days ago
Moreover, if I want to evaluate vision language models in NIAH, is it needed to add system message?
Best regards, Long
Hi! In fact, we have the system prompt in this line.
In this paper, we test only the base models, which do not include the SFT stage. To test the Instruct models, it is necessary to include a system prompt based on their chat templates. For example, Llama3.1-Instruct:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
[YOUR SYSTEM RROMPT]<|eot_id|><|start_header_id|>user<|end_header_id|>
Congrats for the insightful paper!
It seems that in the file test_niah_llama.py, no system message is added to the input before model.generate. I would like to ask whether this omission was intentional or if it is something that needs to be added manually.