Where are the "preceding sentences" from?

[cgt-woailol]

I want to build my own dataset to fine-tune a Critic model, but I found that the prompt in the code has a preceding presence, and I don't know where it comes from and how it was generated?

PROMPT_DICT = { "context": ( "Given an instruction, please make a judgment on whether finding some external documents from the web (e.g., Wikipedia) helps to generate a better response. Please answer [Yes] or [No] and write an explanation.\n\n" "##\nInstruction: Give three tips for staying healthy.\n" "Need retrieval?: [Yes]\n" "Explanation: There might be some online sources listing three tips for staying healthy or some reliable sources to explain the effects of different behaviors on health. So retrieving documents is helpful to improve the response to this query.\n\n" "##\nInstruction: Describe a time when you had to make a difficult decision.\n" "Need retrieval?: [No]\n" "Explanation: This instruction is asking about some personal experience and thus it does not require one to find some external documents.\n\n" "##\nInstruction: Write a short story in third person narration about a protagonist who has to make an important career decision.\n" "Need retrieval?: [No]\n" "Explanation: This instruction asks us to write a short story, which does not require external evidence to verify.\n\n" "##\nInstruction: What is the capital of France?\n" "Need retrieval?: [Yes]\n" "Explanation: While the instruction simply asks us to answer the capital of France, which is a widely known fact, retrieving web documents for this question can still help.\n\n" "##\n Instruction: Find the area of a circle given its radius. Radius = 4\n" "Need retrieval?: [No]\n" "Explanation: This is a math question and although we may be able to find some documents describing a formula, it is unlikely to find a document exactly mentioning the answer.\n\n" "##\nInstruction: Arrange the words in the given sentence to form a grammatically correct sentence. quickly the brown fox jumped\n" "Need retrieval?: [No]\n" "Explanation: This task doesn't require any external evidence, as it is a simple grammatical question.\n\n" "##\nInstruction: Explain the process of cellular respiration in plants." "Need retrieval?: [Yes]\n" "Explanation: This instruction asks for a detailed description of a scientific concept, and is highly likely that we can find a reliable and useful document to support the response.\n\n" "##\nInstruction:{instruction}\n" "Need retrieval?: " ), "multi_retrieval": ( "You will be provided with an instruction, evidence, output sentence, and preceding sentences (optional). If the preceding sentence is given, the output should be the sentence that follows those preceding sentences. Your task is to determine whether the information in the output sentence can be fully verified by the evidence or if it requires further external verification. If the output sentence can be verified solely with the evidence or doesn’t require any verification, respond with [No Retrieval]. If additional information is needed to verify the output sentence, respond with [Retrieval]. Please provide explanations for your judgments.\n\n" "##\nInstruction: Explain the use of word embeddings in Natural Language Processing.\n" "Preceding sentences: Word embeddings are one of the most powerful tools available for Natural Language Processing (NLP). They are mathematical representations of words or phrases in a vector space, allowing similarities between words and the context in which they are used to be measured.\n" "Evidence: Word embedding\nWord embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with one dimension per word to a continuous vector space with a much lower dimension.\n" "Output: Word embeddings are useful for tasks such as sentiment analysis, text classification, predicting the next word in a sequence, and understanding synonyms and analogies.\n" "Rating: [Retrieval]\n" "Explanation: The output discusses the applications of word embeddings, while the evidence only discusses the definitions of word embeddings and how it works. Therefore, we need to retrieve other evidence to verify whether the output is actually correct or not.\n" "###\nInstruction: {instruction}\n" "Preceding sentences: {preceding_sentences}\n" "Evidence: {evidence}\n" "Output: {target_output}\n" "Rating: "), "multi_retrieval_no_preceding": ( "You will be provided with an instruction, evidence, output sentence, and preceding sentences (optional). If the preceding sentence is given, the output should be the sentence that follows those preceding sentences. Your task is to determine whether the information in the output sentence can be fully verified by the evidence or if it requires further external verification. If the output sentence can be verified solely with the evidence or doesn’t require any verification, respond with [No Retrieval]. If additional information is needed to verify the output sentence, respond with [Retrieval]. Please provide explanations for your judgments.\n\n" "##\nInstruction: Explain the use of word embeddings in Natural Language Processing.\n" "Preceding sentences: Word embeddings are one of the most powerful tools available for Natural Language Processing (NLP). They are mathematical representations of words or phrases in a vector space, allowing similarities between words and the context in which they are used to be measured.\n" "Evidence: Word embedding\nWord embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with one dimension per word to a continuous vector space with a much lower dimension.\n" "Output: Word embeddings are useful for tasks such as sentiment analysis, text classification, predicting the next word in a sequence, and understanding synonyms and analogies.\n" "Rating: [Retrieval]\n" "Explanation: The output discusses the applications of word embeddings, while the evidence only discusses the definitions of word embeddings and how it works. Therefore, we need to retrieve other evidence to verify whether the output is actually correct or not.\n" "###\nInstruction: {instruction}\n" "Evidence: {evidence}\n" "Output: {target_output}\n" "Rating: " ), "multi_retrieval_three_way": ( "You will be provided with an instruction, evidence, output sentence, and preceding sentences (optional). If the preceding sentence is given, the output should be the sentence that follows those preceding sentences. Your task is to determine whether the information in the output sentence can be fully verified by the evidence or if it requires further external verification. There are three cases:\n" "- If the output sentence can be verified solely with the evidence, then respond with [Continue to Use Evidence]. \n" "- If the sentence doesn't require any factual verification (e.g., a subjective sentence or a sentence about common sense), then respond with [No Retrieval]. \n" "If additional information is needed to verify the output sentence, respond with [Retrieval]. Please provide explanations for your judgments. \n\n" "##\nInstruction: Explain the use of word embeddings in Natural Language Processing.\n" "Preceding sentences: Word embeddings are one of the most powerful tools available for Natural Language Processing (NLP). They are mathematical representations of words or phrases in a vector space, allowing similarities between words and the context in which they are used to be measured. \n" "Evidence:\nWord embedding\nWord embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with one dimension per word to a continuous vector space with a much lower dimension. \n" "Output: Word embeddings are useful for tasks such as sentiment analysis, text classification, predicting the next word in a sequence, and understanding synonyms and analogies.\n" "Rating: [Retrieval]\n" "Explanation: The output discusses the applications of word embeddings, while the evidence only discusses the definitions of word embeddings and how it works. Therefore, we need to retrieve other evidence to verify whether the output is correct or not.\n" "###\nInstruction: {instruction}\n" "Preceding sentences: {preceding_sentences}\n" "Evidence: {evidence}\n" "Output: {target_output}\n" "Rating: " ), "multi_retrieval_three_way_no_preceding": ( "You will be provided with an instruction, evidence, output sentence, and preceding sentences (optional). If the preceding sentence is given, the output should be the sentence that follows those preceding sentences. Your task is to determine whether the information in the output sentence can be fully verified by the evidence or if it requires further external verification. There are three cases:\n" "- If the output sentence can be verified solely with the evidence, then respond with [Continue to Use Evidence]. \n" "- If the sentence doesn't require any factual verification (e.g., a subjective sentence or a sentence about common sense), then respond with [No Retrieval]. \n" "- If additional information is needed to verify the output sentence, respond with [Retrieval]. Please provide explanations for your judgments. \n\n" "##\nInstruction: Explain the use of word embeddings in Natural Language Processing.\n" "Preceding sentences: Word embeddings are one of the most powerful tools available for Natural Language Processing (NLP). They are mathematical representations of words or phrases in a vector space, allowing similarities between words and the context in which they are used to be measured. \n" "Evidence:\nWord embedding\nWord embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing (NLP) where words or phrases from the vocabulary are mapped to vectors of real numbers. Conceptually it involves a mathematical embedding from a space with one dimension per word to a continuous vector space with a much lower dimension. \n" "Output: Word embeddings are useful for tasks such as sentiment analysis, text classification, predicting the next word in a sequence, and understanding synonyms and analogies.\n" "Rating: [Retrieval]\n" "Explanation: The output discusses the applications of word embeddings, while the evidence only discusses the definitions of word embeddings and how it works. Therefore, we need to retrieve other evidence to verify whether the output is correct or not.\n" "###\nInstruction: {instruction}\n" "Evidence: {evidence}\n" "Output: {target_output}\n" "Rating: " ) }

[AkariAsai]

It will be added when you run the create_retrieval_data.py script! Essentially what we did is to select the sentences before the target output sentence y_t when it makes reflection token predictions!

https://github.com/AkariAsai/self-rag/blob/main/data_creation/generator/create_retrieval_data.py#L89

This discussion might be relevant: https://github.com/AkariAsai/self-rag/issues/7

[cgt-woailol]

当您运行脚本时，它将被添加！从本质上讲，我们所做的是在目标输出句子进行反射标记预测时选择它前面的句子！create_retrieval_data.py``y_t

https://github.com/AkariAsai/self-rag/blob/main/data_creation/generator/create_retrieval_data.py#L89

这个讨论可能是相关的：#7

But I want to use gpt-4 to build my datasets. What you say is the next setp. So I mean that the 'preceding sentences' from chatgpt_need_retrieval.py.

[AkariAsai]

Could you ellaborate your question a bit more? preceding_sentences will be added if you run the create_retrieval_data.py) script with --multiple_sent option.

[AkariAsai]

I am closing this issue for now, but feel free to reopen it!

[Aman-4-Real]

I want to build my own dataset to fine-tune a Critic model, but I found that the prompt in the code has a preceding presence, and I don't know where it comes from and how it was generated?

PROMPT_DICT = { "context": ( "Given an instruction, please make ......TL;DR...... is correct or not.\n" "###\nInstruction: {instruction}\n" "Evidence: {evidence}\n" "Output: {target_output}\n" "Rating: " ) }

I am dealing with the same thing to fine-tune a Critic model. I guess the order in the guideline to run this step https://github.com/AkariAsai/self-rag/blob/ed2296851567e0b656511e09ccbca49360899865/data_creation/generator/README.md?plain=1#L35-L47 may be wrong. Directly using this command will lead to a "preceding presence missing" error as you mentioned.

What the author said here is helpful. So I ran the #L35-L47 after doing "Continuous retrieval (t>1)" and before "Create IsRel and IsSUp input data" and got the final constructed data in the correct format.