search

hkust-nlp / deita

Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]
Apache License 2.0
458 stars 28 forks source link

What content is encoded when Llama13B encoded a sentence #8

Closed cgpeter96 closed 7 months ago

cgpeter96 commented 7 months ago

Only prompt? or prompt+response?

VPeterV commented 7 months ago

We have integrated the prompt within our code. If you want to use our scorers, you only need to give your query sentence for scoring complexity or query + response for scoring quality.

Our code will convert your input (i.e. query or query + response) to our designed format for scorers to score samples.

edbeeching commented 7 months ago

Hi @VPeterV. Related question: In the Score-First, Diversity-Aware Data Selection algorithm are the llama-13b sentence embeddings a mean of the token embeddings, the last token embedding, or something else?

VPeterV commented 7 months ago

Hi @VPeterV. Related question: In the Score-First, Diversity-Aware Data Selection algorithm are the llama-13b sentence embeddings a mean of the token embeddings, the last token embedding, or something else?

Hi @edbeeching ! We encode the sentence by using the hidden states of the last token for one sentence (i.e. the last token in query + response)