danilotpnta / IR2-project

Reproducibility study of the paper: "InPars Toolkit: A Unified and Reproducible Synthetic Data Generation Pipeline for Neural Information Retrieval"

MIT License

1 stars 0 forks source link

Prompt template is not adapted to the dataset #4

Open m-krastev opened 1 week ago

m-krastev commented 1 week ago

Adaptive Prompt Template

Issues:

Currently, the provided templates in prompts are relatively hard-coded for all datasets.
The templates were designed for not very good models. If we use Llama3+, it might be better to at least ask the model to generate more queries.
Would be good if the templates were adapted to the specific dataset, e.g. for webqsp, the template could be "What is the answer to the question: {question}?".

MJHamar commented 6 days ago

can't we just fit a prompt with dspy instead of trying to hard-code new ones?

m-krastev commented 6 days ago

Issue is, it might not be very compute-efficient. We can try to generate one prompt per dataset. I need to say that I am thinking along the lines of using a 70B+ model for generation. If we only use <=8B models, this concern is not as real at all.