Add tools to doc and to prompt

huggingface / lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

MIT License

845 stars 100 forks source link

Add tools to doc and to prompt #321

Closed srossi93 closed 1 month ago

srossi93 commented 2 months ago

This PR adds the possibility of adding tools in the class Doc to be used in the apply_chat_template function.

This PR is just a suggestion but it currently lacks any proper testing.

Example usage

def prompt_fn(sample, task_name: str = None):
  return Doc(
    task_name=task_name,
    query=sample["question"],
    choices=[sample["exact_tool"]],
    gold_index=0,
    instruction="",
    tools=sample["tools"],  # New 
  )

clefourrier commented 2 months ago

Hi!

For the moment, we don't have any task which would require this new feature, and we avoid adding unused code to the code base. Is there a specific associated task you would want which would require this?