search

HKUNLP / STRING

Data and code for our paper "Why Does the Effective Context Length of LLMs Fall Short?"
MIT License
64 stars 3 forks source link

Questions about system prompt in NIAH #5

Open Cooperx521 opened 3 days ago

Cooperx521 commented 3 days ago

Congrats for the insightful paper!

    text_inputs = get_input_ctx_multi(tokenizer=tokenizer, ctx_len=test_max_length, question=question, needles=needles)
    inputs = tokenizer(text_inputs, return_tensors="pt", return_token_type_ids=False).to(model.device)
    prompt_length = inputs.input_ids.size()[-1]
    sample = model.generate(**inputs, do_sample=False, max_new_tokens=max_new_tokens)

It seems that in the file test_niah_llama.py, no system message is added to the input before model.generate. I would like to ask whether this omission was intentional or if it is something that needs to be added manually.

Cooperx521 commented 2 days ago

Moreover, if I want to evaluate vision language models in NIAH, is it needed to add system message?

Best regards, Long