Prompts as Auto-Optimized Training Hyperparameters

xorsuyash commented 2 months ago

Description

Training Best-in-Class IR Models from Scratch with 10 Gold Labels. This method allows for the training of small-scale (under 100M parameters) neural information retrieval (IR) models using as few as 10 gold relevance labels by generating synthetic queries for documents using a language model (LM). PATH is the automatic optimization of the LM prompt based on training quality.

Implementation Details

Integrate a large autoregressive LM (e.g., GPT-3.5-turbo) for synthetic query generation.
Use a small pretrained encoder model (e.g., DeBERTa, MiniLM) for the reranking task.
Sample a subset of the document corpus and Generate synthetic queries using the LM with the provided prompt template.
Sample hard negatives from the corpus to form training triplets.
Train the encoder model on the generated triplets to act as a point-wise reranker.
Evaluate the reranker using a metric such as NDCG@10 on the provided relevance judgments.
Implement an optimization loop where the LM generates new candidate prompt instructions as stated in research paper.
Can use tools from the DSPy framework to automate the prompt optimization process.
Evaluate the reranker trained with each new prompt to find the prompt that maximizes the reranker’s quality.

Domain

Generative AI , NLP

Organisation

SamagraX

Mentors

@xorsuyash

Complexity

Medium

References

PATH

harshaharod21 commented 1 month ago

@xorsuyash I want to work on this issue. I have went through the task and have understood well. While doing the local setup I have a doubt regarding the ansible script, is it optional? Because for that we are required to use Hashicorp vault, which is a paid service.

GautamR-Samagra commented 1 month ago

@xorsuyash I want to work on this issue. I have went through the task and have understood well. While doing the local setup I have a doubt regarding the ansible script, is it optional? Because for that we are required to use Hashicorp vault, which is a paid service.

not sure which ansible script you are referring to here ? can you clarify?

xorsuyash commented 1 month ago

@harshaharod21 You don't need to setup ai-tools for this task, this for experimentation purpose.

harshaharod21 commented 1 month ago

@xorsuyash I want to work on this issue. I have went through the task and have understood well. While doing the local setup I have a doubt regarding the ansible script, is it optional? Because for that we are required to use Hashicorp vault, which is a paid service.

not sure which ansible script you are referring to here ? can you clarify?

Actually that was given in the readme file of the code

harshaharod21 commented 1 month ago

@harshaharod21 You don't need to setup ai-tools for this task, this for experimentation purpose.

Okay thanks, I got it! So anyways, this issue is open to contribution right?

Samagra-Development / ai-tools