Closed lambda7xx closed 2 months ago
dosubot[bot]
commented
6 months ago Hey @lambda7xx! 🚀 Great to see you diving into another interesting challenge. Remember our last adventure? Buckle up because this one looks just as fun! I'm on it and will get back to you with some insights shortly. 🧐✨
logan-markewich
commented
6 months ago Override the LLM class complete() and/ chat() function, insert a timer and call super?
lambda7xx
commented
6 months ago Override the LLM class complete() and/ chat() function, insert a timer and call super?
good idea.
dosubot[bot]
commented
6 months ago Yes, your approach to measure the end-to-end time for each single new query generated in a multi-step query is a good one. You can indeed override the complete() and chat() functions of the LLM class, insert a timer, and call super. Here's how you can implement it:
import time
from llama_index.llms import HuggingFaceLLM
class TimedHuggingFaceLLM(HuggingFaceLLM):
def complete(self, *args, **kwargs):
start_time = time.time()
result = super().complete(*args, **kwargs)
end_time = time.time()
print(f"Query Time: {end_time - start_time} seconds")
return result
def chat(self, *args, **kwargs):
start_time = time.time()
result = super().chat(*args, **kwargs)
end_time = time.time()
print(f"Chat Time: {end_time - start_time} seconds")
return resultYou can then use this TimedHuggingFaceLLM class in your code like this:
llm = TimedHuggingFaceLLM(
model_name="mistralai/Mistral-7B-Instruct-v0.1",
tokenizer_name="mistralai/Mistral-7B-Instruct-v0.1",
query_wrapper_prompt=PromptTemplate("<s>[INST] {query_str} [/INST] </s>\n"),
context_window=3900,
max_new_tokens=256,
model_kwargs={"torch_dtype": torch.bfloat16},
generate_kwargs={"temperature": 0.2, "top_k": 5, "top_p": 0.95},
device_map="auto",
)
step_decompose_transform = StepDecomposeQueryTransform(llm=Settings.llm, verbose=True)
# Rest of your code...This way, every time a new query is generated, the time it took will be printed to the console. This should give you a good idea of the performance of your queries.
Please note that the chat() method is not explicitly defined in the HuggingFaceLLM class in the LlamaIndex library. The functionality related to chat interactions might be handled differently or is not included within the scope of the provided code excerpt. Given the context, it's possible that chat functionality, if present, would leverage the model's capabilities to generate responses in a conversational context, potentially using the _tokenizer_messages_to_prompt method to format chat messages into a suitable prompt for the model. However, without a direct reference to a chat() method in the provided code, we cannot specify its functionalities based on the given context.
For more information, you can refer to the LlamaIndex documentation and the source code of the HuggingFaceLLM class.
lambda7xx
commented
6 months ago Yes, your approach to measure the end-to-end time for each single new query generated in a multi-step query is a good one. You can indeed override the
complete()andchat()functions of the LLM class, insert a timer, and callsuper. Here's how you can implement it:import time from llama_index.llms import HuggingFaceLLM class TimedHuggingFaceLLM(HuggingFaceLLM): def complete(self, *args, **kwargs): start_time = time.time() result = super().complete(*args, **kwargs) end_time = time.time() print(f"Query Time: {end_time - start_time} seconds") return result def chat(self, *args, **kwargs): start_time = time.time() result = super().chat(*args, **kwargs) end_time = time.time() print(f"Chat Time: {end_time - start_time} seconds") return resultYou can then use this
TimedHuggingFaceLLMclass in your code like this:llm = TimedHuggingFaceLLM( model_name="mistralai/Mistral-7B-Instruct-v0.1", tokenizer_name="mistralai/Mistral-7B-Instruct-v0.1", query_wrapper_prompt=PromptTemplate("<s>[INST] {query_str} [/INST] </s>\n"), context_window=3900, max_new_tokens=256, model_kwargs={"torch_dtype": torch.bfloat16}, generate_kwargs={"temperature": 0.2, "top_k": 5, "top_p": 0.95}, device_map="auto", ) step_decompose_transform = StepDecomposeQueryTransform(llm=Settings.llm, verbose=True) # Rest of your code...This way, every time a new query is generated, the time it took will be printed to the console. This should give you a good idea of the performance of your queries.
Please note that the
chat()method is not explicitly defined in theHuggingFaceLLMclass in the LlamaIndex library. The functionality related to chat interactions might be handled differently or is not included within the scope of the provided code excerpt. Given the context, it's possible that chat functionality, if present, would leverage the model's capabilities to generate responses in a conversational context, potentially using the_tokenizer_messages_to_promptmethod to format chat messages into a suitable prompt for the model. However, without a direct reference to achat()method in the provided code, we cannot specify its functionalities based on the given context.For more information, you can refer to the LlamaIndex documentation and the source code of the
HuggingFaceLLMclass.Sources
- docs/docs/understanding/putting_it_all_together/q_and_a.md
- llama-index-integrations/llms/llama-index-llms-huggingface/llama_index/llms/huggingface/base.py
- llama-index-core/llama_index/core/query_engine/multistep_query_engine.py
About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
that's awesome, thank you
dosubot[bot]
commented
6 months ago @lambda7xx You're welcome! I'm glad you found the information helpful. If you have any more questions or need further assistance with your project, feel free to ask. Good luck with measuring the query latency, and happy coding!
lambda7xx
commented
6 months ago i use the code to run one query.
import torch
# from transformers import BitsAndBytesConfig
from llama_index.llms.huggingface import HuggingFaceLLM
from llama_index.core.prompts import PromptTemplate
from llama_index.core import Settings
from llama_index.core import VectorStoreIndex
from llama_index.core import SimpleDirectoryReader
from llama_index.core.response.notebook_utils import display_response
from llama_index.core.indices.query.query_transform.base import (
StepDecomposeQueryTransform,
)
import llama_index.core
import time
import time
# from llama_index.llms import HuggingFaceLLM
llama_index.core.set_global_handler("simple")
all_time = []
class TimedHuggingFaceLLM(HuggingFaceLLM):
def complete(self, *args, **kwargs):
start_time = time.time()
result = super().complete(*args, **kwargs)
end_time = time.time()
# print(f"args[query]:{args} and **kwargs:{kwargs}")
#print(f"\n\n\n Query Time: {end_time - start_time} seconds and result:{result}")
print(f"\n\n\n Query Time: {end_time - start_time} seconds ")
print("--------\n\n\n")
return result
def chat(self, *args, **kwargs):
start_time = time.time()
result = super().chat(*args, **kwargs)
end_time = time.time()
print(f"Chat Time: {end_time - start_time} seconds")
return result
llm = TimedHuggingFaceLLM(
model_name="mistralai/Mistral-7B-Instruct-v0.1",
tokenizer_name="mistralai/Mistral-7B-Instruct-v0.1",
query_wrapper_prompt=PromptTemplate("<s>[INST] {query_str} [/INST] </s>\n"),
context_window=3900,
max_new_tokens=256,
model_kwargs={"torch_dtype": torch.bfloat16},
generate_kwargs={"temperature": 0.2, "top_k": 5, "top_p": 0.95},
device_map="auto",
)
# recordMemory("base model llm ")
Settings.llm = llm
llama_survey_path = "../data/survery/llm_survery_paper.json"
import json
origin_queries = []
with open(llama_survey_path) as f:
data = json.load(f)
i = 0
for key in data['examples']:
if i < 1:
origin_queries.append(key['query'])
i = i + 1
embeddings = ["local:BAAI/bge-small-en-v1.5", "local:BAAI/bge-large-en-v1.5", "local:thenlper/gte-base", "local:thenlper/gte-large"]
embeddings = ["local:BAAI/bge-small-en-v1.5", "local:thenlper/gte-base"]
embeddings = ["local:BAAI/bge-small-en-v1.5"]
filename = "llama_survey_default_chunk_size_multi_step_query.csv"
step_decompose_transform = StepDecomposeQueryTransform(llm=Settings.llm, verbose=True)
from llama_index.core.query_engine import MultiStepQueryEngine
for query in origin_queries:
for em in embeddings:
print(f"******query:{query}\n\n")
Settings.embed_model = em
start = time.time()
documents = SimpleDirectoryReader("../data/survery/").load_data() #data/llmama2_paper.json ./data/survery/llm_survery_paper.json
end = time.time()
load_time = end - start
start = time.time()
index = VectorStoreIndex.from_documents(documents=documents)
end = time.time()
index_time = end - start
query_engine = index.as_query_engine()
start = time.time()
query_engine = MultiStepQueryEngine(
query_engine=query_engine,
query_transform=step_decompose_transform)
end = time.time()
print(f" MultiStepQueryEngine, duration:{end - start}")
start = time.time()
response = query_engine.query(query)
end = time.time()
print(f" query_engine.query, duration:{end - start}")
sub_qa = response.metadata["sub_qa"]
tuples = [(t[0], t[1].response) for t in sub_qa]
print(f"len(tuples): {len(tuples)}")
#print(tuples)
for t in tuples:
print("*********",t[1],'\n\n')my log
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:01<00:01, 1.30s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00, 1.07it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00, 1.01it/s]
/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:492: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.2` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:497: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
warnings.warn(
/opt/conda/envs/pytorch/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:509: UserWarning: `do_sample` is set to `False`. However, `top_k` is set to `5` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_k`.
warnings.warn(
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
******query:What are the potential risks associated with large language models (LLMs) according to the context information?
MultiStepQueryEngine, duration:5.030632019042969e-05
** Prompt: **
<s>[INST] The original question is as follows: What are the potential risks associated with large language models (LLMs) according to the context information?
We have an opportunity to answer some, or all of the question from a knowledge source. Context information for the knowledge source is provided below, as well as previous reasoning steps.
Given the context and previous reasoning, return a question that can be answered from the context. This question can be the same as the original question, or this question can represent a subcomponent of the overall question.It should not be irrelevant to the original question.
If we cannot extract more information from the context, provide 'None' as the answer. Some examples are given below:
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides names of the winners of the 2020 Australian Open
Previous reasoning: None
Next question: Who was the winner of the 2020 Australian Open?
Question: Who was the winner of the 2020 Australian Open?
Knowledge source context: Provides names of the winners of the 2020 Australian Open
Previous reasoning: None.
New question: Who was the winner of the 2020 Australian Open?
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides information about the winners of the 2020 Australian Open
Previous reasoning:
- Who was the winner of the 2020 Australian Open?
- The winner of the 2020 Australian Open was Novak Djokovic.
New question: None
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides information about the winners of the 2020 Australian Open - includes biographical information for each winner
Previous reasoning:
- Who was the winner of the 2020 Australian Open?
- The winner of the 2020 Australian Open was Novak Djokovic.
New question: How many Grand Slam titles does Novak Djokovic have?
Question: What are the potential risks associated with large language models (LLMs) according to the context information?
Knowledge source context: None
Previous reasoning: None
New question: [/INST] </s>
**************************************************
** Completion: **
What are the potential risks associated with large language models (LLMs) according to the context information?
**************************************************
Query Time: 1.084970235824585 seconds
--------
[1;3;33m> Current query: What are the potential risks associated with large language models (LLMs) according to the context information?
[0m[1;3;38;5;200m> New query:
What are the potential risks associated with large language models (LLMs) according to the context information?
[0m** Prompt: **
<s>[INST] Context information is below.
---------------------
file_path: /home/ubuntu/llama-index/multi_step_query/../data/survery/llm_survery_paper.json
{
"examples": [
{
"query": "What are the potential risks associated with large language models (LLMs) according to the context information?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Evaluating Large Language Models: A\nComprehensive Survey\nZishan Guo\u2217, Renren Jin\u2217, Chuang Liu\u2217, Yufei Huang, Dan Shi, Supryadi\nLinhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong\u2020\nTianjin University\n{guozishan, rrjin, liuc_09, yuki_731, shidan, supryadi}@tju.edu.cn\n{linhaoyu, yan_liu, jiaxuanlee, xbj1355, dyxiong}@tju.edu.cn\nAbstract\nLarge language models (LLMs) have demonstrated remarkable capabilities\nacross a broad spectrum of tasks. They have attracted significant attention\nand been deployed in numerous downstream applications. Nevertheless, akin\nto a double-edged sword, LLMs also present potential risks. They could\nsuffer from private data leaks or yield inappropriate, harmful, or misleading\ncontent. Additionally, the rapid progress of LLMs raises concerns about the\npotential emergence of superintelligent systems without adequate safeguards.\nTo effectively capitalize on LLM capacities as well as ensure their safe and\nbeneficial development, it is critical to conduct a rigorous and comprehensive\nevaluation of LLMs.\nThis survey endeavors to offer a panoramic perspective on the evaluation\nof LLMs. We categorize the evaluation of LLMs into three major groups:\nknowledgeandcapabilityevaluation, alignmentevaluationandsafetyevaluation.\nIn addition to the comprehensive review on the evaluation methodologies and\nbenchmarks on these three aspects, we collate a compendium of evaluations\npertaining to LLMs\u2019 performance in specialized domains, and discuss the\nconstruction of comprehensive evaluation platforms that cover LLM evaluations\non capabilities, alignment, safety, and applicability.\nWe hope that this comprehensive overview will stimulate further research\ninterests in the evaluation of LLMs, with the ultimate goal of making evaluation\nserve as a cornerstone in guiding the responsible development of LLMs. We\nenvision that this will channel their evolution into a direction that maximizes\nsocietal benefit while minimizing potential risks. A curated list of related\npapers has been publicly available at a GitHub repository.1\n\u2217Equal contribution\n\u2020Corresponding author.\n1https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers\n1arXiv:2310.19736v3 [cs.CL] 25 Nov 2023"
],
"reference_answer": "According to the context information, the potential risks associated with large language models (LLMs) include the following:\n\n1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.\n\n2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.\n\n3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.\n\nTo ensure the safe and beneficial development of LLMs, it is crucial to conduct rigorous and comprehensive evaluations of their capabilities, alignment with human values, and safety measures.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "How does the survey categorize the evaluation of LLMs and what are the three major groups mentioned?
file_path: /home/ubuntu/llama-index/multi_step_query/../data/survery/llm_survery_paper.json
Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.\n\n3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.\n\nTo ensure the safe and beneficial development of LLMs, it is crucial to conduct rigorous and comprehensive evaluations of their capabilities, alignment with human values, and safety measures.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "How does the survey categorize the evaluation of LLMs and what are the three major groups mentioned?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Evaluating Large Language Models: A\nComprehensive Survey\nZishan Guo\u2217, Renren Jin\u2217, Chuang Liu\u2217, Yufei Huang, Dan Shi, Supryadi\nLinhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong\u2020\nTianjin University\n{guozishan, rrjin, liuc_09, yuki_731, shidan, supryadi}@tju.edu.cn\n{linhaoyu, yan_liu, jiaxuanlee, xbj1355, dyxiong}@tju.edu.cn\nAbstract\nLarge language models (LLMs) have demonstrated remarkable capabilities\nacross a broad spectrum of tasks. They have attracted significant attention\nand been deployed in numerous downstream applications. Nevertheless, akin\nto a double-edged sword, LLMs also present potential risks. They could\nsuffer from private data leaks or yield inappropriate, harmful, or misleading\ncontent. Additionally, the rapid progress of LLMs raises concerns about the\npotential emergence of superintelligent systems without adequate safeguards.\nTo effectively capitalize on LLM capacities as well as ensure their safe and\nbeneficial development, it is critical to conduct a rigorous and comprehensive\nevaluation of LLMs.\nThis survey endeavors to offer a panoramic perspective on the evaluation\nof LLMs. We categorize the evaluation of LLMs into three major groups:\nknowledgeandcapabilityevaluation, alignmentevaluationandsafetyevaluation.\nIn addition to the comprehensive review on the evaluation methodologies and\nbenchmarks on these three aspects, we collate a compendium of evaluations\npertaining to LLMs\u2019 performance in specialized domains, and discuss the\nconstruction of comprehensive evaluation platforms that cover LLM evaluations\non capabilities, alignment, safety, and applicability.\nWe hope that this comprehensive overview will stimulate further research\ninterests in the evaluation of LLMs, with the ultimate goal of making evaluation\nserve as a cornerstone in guiding the responsible development of LLMs. We\nenvision that this will channel their evolution into a direction that maximizes\nsocietal benefit while minimizing potential risks. A curated list of related\npapers has been publicly available at a GitHub repository.1\n\u2217Equal contribution\n\u2020Corresponding author.\n1https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers\n1arXiv:2310.19736v3 [cs.CL] 25 Nov 2023"
],
"reference_answer": "The survey categorizes the evaluation of LLMs into three major groups: knowledge and capability evaluation, alignment evaluation, and safety evaluation.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "What are the different types of reasoning discussed in the document?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Contents\n1 Introduction 4\n2 Taxonomy and Roadmap 6\n3 Knowledge and Capability Evaluation 8\n3.1 Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . .
---------------------
Given the context information and not prior knowledge, answer the query.
Query:
What are the potential risks associated with large language models (LLMs) according to the context information?
Answer: [/INST] </s>
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
**************************************************
** Completion: **
The potential risks associated with large language models (LLMs) according to the context information are:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
**************************************************
Query Time: 7.76639461517334 seconds
--------
** Prompt: **
<s>[INST] The original question is as follows: What are the potential risks associated with large language models (LLMs) according to the context information?
We have an opportunity to answer some, or all of the question from a knowledge source. Context information for the knowledge source is provided below, as well as previous reasoning steps.
Given the context and previous reasoning, return a question that can be answered from the context. This question can be the same as the original question, or this question can represent a subcomponent of the overall question.It should not be irrelevant to the original question.
If we cannot extract more information from the context, provide 'None' as the answer. Some examples are given below:
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides names of the winners of the 2020 Australian Open
Previous reasoning: None
Next question: Who was the winner of the 2020 Australian Open?
Question: Who was the winner of the 2020 Australian Open?
Knowledge source context: Provides names of the winners of the 2020 Australian Open
Previous reasoning: None.
New question: Who was the winner of the 2020 Australian Open?
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides information about the winners of the 2020 Australian Open
Previous reasoning:
- Who was the winner of the 2020 Australian Open?
- The winner of the 2020 Australian Open was Novak Djokovic.
New question: None
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides information about the winners of the 2020 Australian Open - includes biographical information for each winner
Previous reasoning:
- Who was the winner of the 2020 Australian Open?
- The winner of the 2020 Australian Open was Novak Djokovic.
New question: How many Grand Slam titles does Novak Djokovic have?
Question: What are the potential risks associated with large language models (LLMs) according to the context information?
Knowledge source context: None
Previous reasoning:
-
What are the potential risks associated with large language models (LLMs) according to the context information?
-
The potential risks associated with large language models (LLMs) according to the context information are:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
New question: [/INST] </s>
**************************************************
** Completion: **
What are the potential risks associated with large language models (LLMs) according to the context information?
**************************************************
Query Time: 0.9940519332885742 seconds
--------
[1;3;33m> Current query: What are the potential risks associated with large language models (LLMs) according to the context information?
[0m[1;3;38;5;200m> New query: What are the potential risks associated with large language models (LLMs) according to the context information?
[0m** Prompt: **
<s>[INST] Context information is below.
---------------------
file_path: /home/ubuntu/llama-index/multi_step_query/../data/survery/llm_survery_paper.json
{
"examples": [
{
"query": "What are the potential risks associated with large language models (LLMs) according to the context information?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Evaluating Large Language Models: A\nComprehensive Survey\nZishan Guo\u2217, Renren Jin\u2217, Chuang Liu\u2217, Yufei Huang, Dan Shi, Supryadi\nLinhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong\u2020\nTianjin University\n{guozishan, rrjin, liuc_09, yuki_731, shidan, supryadi}@tju.edu.cn\n{linhaoyu, yan_liu, jiaxuanlee, xbj1355, dyxiong}@tju.edu.cn\nAbstract\nLarge language models (LLMs) have demonstrated remarkable capabilities\nacross a broad spectrum of tasks. They have attracted significant attention\nand been deployed in numerous downstream applications. Nevertheless, akin\nto a double-edged sword, LLMs also present potential risks. They could\nsuffer from private data leaks or yield inappropriate, harmful, or misleading\ncontent. Additionally, the rapid progress of LLMs raises concerns about the\npotential emergence of superintelligent systems without adequate safeguards.\nTo effectively capitalize on LLM capacities as well as ensure their safe and\nbeneficial development, it is critical to conduct a rigorous and comprehensive\nevaluation of LLMs.\nThis survey endeavors to offer a panoramic perspective on the evaluation\nof LLMs. We categorize the evaluation of LLMs into three major groups:\nknowledgeandcapabilityevaluation, alignmentevaluationandsafetyevaluation.\nIn addition to the comprehensive review on the evaluation methodologies and\nbenchmarks on these three aspects, we collate a compendium of evaluations\npertaining to LLMs\u2019 performance in specialized domains, and discuss the\nconstruction of comprehensive evaluation platforms that cover LLM evaluations\non capabilities, alignment, safety, and applicability.\nWe hope that this comprehensive overview will stimulate further research\ninterests in the evaluation of LLMs, with the ultimate goal of making evaluation\nserve as a cornerstone in guiding the responsible development of LLMs. We\nenvision that this will channel their evolution into a direction that maximizes\nsocietal benefit while minimizing potential risks. A curated list of related\npapers has been publicly available at a GitHub repository.1\n\u2217Equal contribution\n\u2020Corresponding author.\n1https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers\n1arXiv:2310.19736v3 [cs.CL] 25 Nov 2023"
],
"reference_answer": "According to the context information, the potential risks associated with large language models (LLMs) include the following:\n\n1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.\n\n2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.\n\n3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.\n\nTo ensure the safe and beneficial development of LLMs, it is crucial to conduct rigorous and comprehensive evaluations of their capabilities, alignment with human values, and safety measures.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "How does the survey categorize the evaluation of LLMs and what are the three major groups mentioned?
file_path: /home/ubuntu/llama-index/multi_step_query/../data/survery/llm_survery_paper.json
Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.\n\n3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.\n\nTo ensure the safe and beneficial development of LLMs, it is crucial to conduct rigorous and comprehensive evaluations of their capabilities, alignment with human values, and safety measures.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "How does the survey categorize the evaluation of LLMs and what are the three major groups mentioned?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Evaluating Large Language Models: A\nComprehensive Survey\nZishan Guo\u2217, Renren Jin\u2217, Chuang Liu\u2217, Yufei Huang, Dan Shi, Supryadi\nLinhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong\u2020\nTianjin University\n{guozishan, rrjin, liuc_09, yuki_731, shidan, supryadi}@tju.edu.cn\n{linhaoyu, yan_liu, jiaxuanlee, xbj1355, dyxiong}@tju.edu.cn\nAbstract\nLarge language models (LLMs) have demonstrated remarkable capabilities\nacross a broad spectrum of tasks. They have attracted significant attention\nand been deployed in numerous downstream applications. Nevertheless, akin\nto a double-edged sword, LLMs also present potential risks. They could\nsuffer from private data leaks or yield inappropriate, harmful, or misleading\ncontent. Additionally, the rapid progress of LLMs raises concerns about the\npotential emergence of superintelligent systems without adequate safeguards.\nTo effectively capitalize on LLM capacities as well as ensure their safe and\nbeneficial development, it is critical to conduct a rigorous and comprehensive\nevaluation of LLMs.\nThis survey endeavors to offer a panoramic perspective on the evaluation\nof LLMs. We categorize the evaluation of LLMs into three major groups:\nknowledgeandcapabilityevaluation, alignmentevaluationandsafetyevaluation.\nIn addition to the comprehensive review on the evaluation methodologies and\nbenchmarks on these three aspects, we collate a compendium of evaluations\npertaining to LLMs\u2019 performance in specialized domains, and discuss the\nconstruction of comprehensive evaluation platforms that cover LLM evaluations\non capabilities, alignment, safety, and applicability.\nWe hope that this comprehensive overview will stimulate further research\ninterests in the evaluation of LLMs, with the ultimate goal of making evaluation\nserve as a cornerstone in guiding the responsible development of LLMs. We\nenvision that this will channel their evolution into a direction that maximizes\nsocietal benefit while minimizing potential risks. A curated list of related\npapers has been publicly available at a GitHub repository.1\n\u2217Equal contribution\n\u2020Corresponding author.\n1https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers\n1arXiv:2310.19736v3 [cs.CL] 25 Nov 2023"
],
"reference_answer": "The survey categorizes the evaluation of LLMs into three major groups: knowledge and capability evaluation, alignment evaluation, and safety evaluation.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "What are the different types of reasoning discussed in the document?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Contents\n1 Introduction 4\n2 Taxonomy and Roadmap 6\n3 Knowledge and Capability Evaluation 8\n3.1 Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . .
---------------------
Given the context information and not prior knowledge, answer the query.
Query: What are the potential risks associated with large language models (LLMs) according to the context information?
Answer: [/INST] </s>
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
**************************************************
** Completion: **
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
**************************************************
Query Time: 9.029383897781372 seconds
--------
** Prompt: **
<s>[INST] The original question is as follows: What are the potential risks associated with large language models (LLMs) according to the context information?
We have an opportunity to answer some, or all of the question from a knowledge source. Context information for the knowledge source is provided below, as well as previous reasoning steps.
Given the context and previous reasoning, return a question that can be answered from the context. This question can be the same as the original question, or this question can represent a subcomponent of the overall question.It should not be irrelevant to the original question.
If we cannot extract more information from the context, provide 'None' as the answer. Some examples are given below:
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides names of the winners of the 2020 Australian Open
Previous reasoning: None
Next question: Who was the winner of the 2020 Australian Open?
Question: Who was the winner of the 2020 Australian Open?
Knowledge source context: Provides names of the winners of the 2020 Australian Open
Previous reasoning: None.
New question: Who was the winner of the 2020 Australian Open?
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides information about the winners of the 2020 Australian Open
Previous reasoning:
- Who was the winner of the 2020 Australian Open?
- The winner of the 2020 Australian Open was Novak Djokovic.
New question: None
Question: How many Grand Slam titles does the winner of the 2020 Australian Open have?
Knowledge source context: Provides information about the winners of the 2020 Australian Open - includes biographical information for each winner
Previous reasoning:
- Who was the winner of the 2020 Australian Open?
- The winner of the 2020 Australian Open was Novak Djokovic.
New question: How many Grand Slam titles does Novak Djokovic have?
Question: What are the potential risks associated with large language models (LLMs) according to the context information?
Knowledge source context: None
Previous reasoning:
-
What are the potential risks associated with large language models (LLMs) according to the context information?
-
The potential risks associated with large language models (LLMs) according to the context information are:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
- What are the potential risks associated with large language models (LLMs) according to the context information?
-
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
New question: [/INST] </s>
**************************************************
** Completion: **
What are the potential risks associated with large language models (LLMs) according to the context information?
**************************************************
Query Time: 1.0837998390197754 seconds
--------
[1;3;33m> Current query: What are the potential risks associated with large language models (LLMs) according to the context information?
[0m[1;3;38;5;200m> New query: What are the potential risks associated with large language models (LLMs) according to the context information?
[0m** Prompt: **
<s>[INST] Context information is below.
---------------------
file_path: /home/ubuntu/llama-index/multi_step_query/../data/survery/llm_survery_paper.json
{
"examples": [
{
"query": "What are the potential risks associated with large language models (LLMs) according to the context information?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Evaluating Large Language Models: A\nComprehensive Survey\nZishan Guo\u2217, Renren Jin\u2217, Chuang Liu\u2217, Yufei Huang, Dan Shi, Supryadi\nLinhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong\u2020\nTianjin University\n{guozishan, rrjin, liuc_09, yuki_731, shidan, supryadi}@tju.edu.cn\n{linhaoyu, yan_liu, jiaxuanlee, xbj1355, dyxiong}@tju.edu.cn\nAbstract\nLarge language models (LLMs) have demonstrated remarkable capabilities\nacross a broad spectrum of tasks. They have attracted significant attention\nand been deployed in numerous downstream applications. Nevertheless, akin\nto a double-edged sword, LLMs also present potential risks. They could\nsuffer from private data leaks or yield inappropriate, harmful, or misleading\ncontent. Additionally, the rapid progress of LLMs raises concerns about the\npotential emergence of superintelligent systems without adequate safeguards.\nTo effectively capitalize on LLM capacities as well as ensure their safe and\nbeneficial development, it is critical to conduct a rigorous and comprehensive\nevaluation of LLMs.\nThis survey endeavors to offer a panoramic perspective on the evaluation\nof LLMs. We categorize the evaluation of LLMs into three major groups:\nknowledgeandcapabilityevaluation, alignmentevaluationandsafetyevaluation.\nIn addition to the comprehensive review on the evaluation methodologies and\nbenchmarks on these three aspects, we collate a compendium of evaluations\npertaining to LLMs\u2019 performance in specialized domains, and discuss the\nconstruction of comprehensive evaluation platforms that cover LLM evaluations\non capabilities, alignment, safety, and applicability.\nWe hope that this comprehensive overview will stimulate further research\ninterests in the evaluation of LLMs, with the ultimate goal of making evaluation\nserve as a cornerstone in guiding the responsible development of LLMs. We\nenvision that this will channel their evolution into a direction that maximizes\nsocietal benefit while minimizing potential risks. A curated list of related\npapers has been publicly available at a GitHub repository.1\n\u2217Equal contribution\n\u2020Corresponding author.\n1https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers\n1arXiv:2310.19736v3 [cs.CL] 25 Nov 2023"
],
"reference_answer": "According to the context information, the potential risks associated with large language models (LLMs) include the following:\n\n1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.\n\n2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.\n\n3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.\n\nTo ensure the safe and beneficial development of LLMs, it is crucial to conduct rigorous and comprehensive evaluations of their capabilities, alignment with human values, and safety measures.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "How does the survey categorize the evaluation of LLMs and what are the three major groups mentioned?
file_path: /home/ubuntu/llama-index/multi_step_query/../data/survery/llm_survery_paper.json
Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.\n\n3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.\n\nTo ensure the safe and beneficial development of LLMs, it is crucial to conduct rigorous and comprehensive evaluations of their capabilities, alignment with human values, and safety measures.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "How does the survey categorize the evaluation of LLMs and what are the three major groups mentioned?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Evaluating Large Language Models: A\nComprehensive Survey\nZishan Guo\u2217, Renren Jin\u2217, Chuang Liu\u2217, Yufei Huang, Dan Shi, Supryadi\nLinhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong\u2020\nTianjin University\n{guozishan, rrjin, liuc_09, yuki_731, shidan, supryadi}@tju.edu.cn\n{linhaoyu, yan_liu, jiaxuanlee, xbj1355, dyxiong}@tju.edu.cn\nAbstract\nLarge language models (LLMs) have demonstrated remarkable capabilities\nacross a broad spectrum of tasks. They have attracted significant attention\nand been deployed in numerous downstream applications. Nevertheless, akin\nto a double-edged sword, LLMs also present potential risks. They could\nsuffer from private data leaks or yield inappropriate, harmful, or misleading\ncontent. Additionally, the rapid progress of LLMs raises concerns about the\npotential emergence of superintelligent systems without adequate safeguards.\nTo effectively capitalize on LLM capacities as well as ensure their safe and\nbeneficial development, it is critical to conduct a rigorous and comprehensive\nevaluation of LLMs.\nThis survey endeavors to offer a panoramic perspective on the evaluation\nof LLMs. We categorize the evaluation of LLMs into three major groups:\nknowledgeandcapabilityevaluation, alignmentevaluationandsafetyevaluation.\nIn addition to the comprehensive review on the evaluation methodologies and\nbenchmarks on these three aspects, we collate a compendium of evaluations\npertaining to LLMs\u2019 performance in specialized domains, and discuss the\nconstruction of comprehensive evaluation platforms that cover LLM evaluations\non capabilities, alignment, safety, and applicability.\nWe hope that this comprehensive overview will stimulate further research\ninterests in the evaluation of LLMs, with the ultimate goal of making evaluation\nserve as a cornerstone in guiding the responsible development of LLMs. We\nenvision that this will channel their evolution into a direction that maximizes\nsocietal benefit while minimizing potential risks. A curated list of related\npapers has been publicly available at a GitHub repository.1\n\u2217Equal contribution\n\u2020Corresponding author.\n1https://github.com/tjunlp-lab/Awesome-LLMs-Evaluation-Papers\n1arXiv:2310.19736v3 [cs.CL] 25 Nov 2023"
],
"reference_answer": "The survey categorizes the evaluation of LLMs into three major groups: knowledge and capability evaluation, alignment evaluation, and safety evaluation.",
"reference_answer_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
}
},
{
"query": "What are the different types of reasoning discussed in the document?",
"query_by": {
"model_name": "gpt-3.5-turbo",
"type": "ai"
},
"reference_contexts": [
"Contents\n1 Introduction 4\n2 Taxonomy and Roadmap 6\n3 Knowledge and Capability Evaluation 8\n3.1 Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . .
---------------------
Given the context information and not prior knowledge, answer the query.
Query: What are the potential risks associated with large language models (LLMs) according to the context information?
Answer: [/INST] </s>
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
**************************************************
** Completion: **
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
**************************************************
Query Time: 9.027866840362549 seconds
--------
** Prompt: **
<s>[INST] Context information is below.
---------------------
Question:
What are the potential risks associated with large language models (LLMs) according to the context information?
Answer:
The potential risks associated with large language models (LLMs) according to the context information are:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
Question: What are the potential risks associated with large language models (LLMs) according to the context information?
Answer:
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
Question: What are the potential risks associated with large language models (LLMs) according to the context information?
Answer:
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
---------------------
Given the context information and not prior knowledge, answer the query.
Query: What are the potential risks associated with large language models (LLMs) according to the context information?
Answer: [/INST] </s>
**************************************************
** Completion: **
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
**************************************************
Query Time: 7.350822448730469 seconds
--------
query_engine.query, duration:36.415526151657104
len(tuples): 3
*********
The potential risks associated with large language models (LLMs) according to the context information are:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
*********
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
*********
The potential risks associated with large language models (LLMs) according to the context information are as follows:
1. Private data leaks: LLMs have the potential to suffer from private data leaks, which can compromise the confidentiality and privacy of sensitive information.
2. Inappropriate, harmful, or misleading content: LLMs can generate content that may be inappropriate, harmful, or misleading. This can have negative consequences in various applications, such as generating biased or offensive language.
3. Emergence of superintelligent systems without safeguards: The rapid progress of LLMs raises concerns about the potential emergence of superintelligent systems that lack adequate safeguards. This can lead to unforeseen and potentially dangerous behavior.
These risks highlight the importance of conducting rigorous and comprehensive evaluations of LLMs to ensure their safe and beneficial development.
I found there are multiple llm call. I don't undestand why we need multiple llm call. first, we generate multiple new query. This is one llm call then we have one llm call for each new query. In my code, there are 3 new query, so the total llm call should be 3+1 = 4, but the log has 7 llm call. I think the 2nd, 4th, 6th and 7th LLM call use the LLM model and retrieve data to generate answer. but what does the 1st, 3th, and 5th LLM call do ?
logan-markewich
commented
6 months ago You are using a step decompose query transform
So it's taking the original query and decomposing it into multiple
The other queries are because it's hitting a refine step. If the retrieved context for a single query is too big to fit into a single LLM call, it has to make multiple LLM calls to refine an answer, so that the LLM can read all the text.
lambda7xx
commented
6 months ago You are using a step decompose query transform
So it's taking the original query and decomposing it into multiple
The other queries are because it's hitting a refine step. If the retrieved context for a single query is too big to fit into a single LLM call, it has to make multiple LLM calls to refine an answer, so that the LLM can read all the text.
so for the 1st, 3th, and 5th LLM call is to refine an answer?
Question Validation
Question
the below is my code.
it will generate multiple new query and return multiple response. If I want to measure the query e2e time for each single new query? how to measure ?