hkust-nlp / deita

Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]
Apache License 2.0
502 stars 27 forks source link

What content is encoded when Llama13B encoded a sentence #8

Closed cgpeter96 closed 10 months ago

cgpeter96 commented 10 months ago

Only prompt? or prompt+response?

VPeterV commented 10 months ago

We have integrated the prompt within our code. If you want to use our scorers, you only need to give your query sentence for scoring complexity or query + response for scoring quality.

Our code will convert your input (i.e. query or query + response) to our designed format for scorers to score samples.

edbeeching commented 10 months ago

Hi @VPeterV. Related question: In the Score-First, Diversity-Aware Data Selection algorithm are the llama-13b sentence embeddings a mean of the token embeddings, the last token embedding, or something else?

VPeterV commented 10 months ago

Hi @VPeterV. Related question: In the Score-First, Diversity-Aware Data Selection algorithm are the llama-13b sentence embeddings a mean of the token embeddings, the last token embedding, or something else?

Hi @edbeeching ! We encode the sentence by using the hidden states of the last token for one sentence (i.e. the last token in query + response)