[Question]: response from query_engine is generating wrong score.

[adityamity]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

When i generate response from query_engine using below code:

`response = query_engine.query(query)`

I get the reponse. However, i see that the node with correct answer is have low score and the one with not that relevant answer with high score.

below is my response looks like:

Response(response='Response 1: My response.', source_nodes=[NodeWithScore(node=TextNode(id_='eb16198b-027e-4d20-9817-52f16c876a49', embedding=None, metadata={'page_label': '12', 'file_name': '/code/Users/docs/sample.pdf', 'document_title': 'Products Sales Prices', excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='//code/Users/docs/sample.pdf', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '12', 'file_name': '/docs/sample.pdf'}, hash='43df66ba0d6e47d395479708b8bda6080f036d7949ec2792493282e992e3a7e8')}, text=\n Reference: \ \n Approver: S. Amblard \n- DCF Page: 12/18 \nEntity : DCF/FI Elastomer Products Sales Prices Definition \nProcedure Type of document: \nNorm Publication date: \n23/04/2021 \n \nArchiving period: WA+10 Printed copies of this document are not managed Classification: D3 \n \nArchive: \nData will be archived by Finance responsible. \n \n3.4 TRANSPORTATION COST \nDefinition : \nTransportation costs remunerate the transport of the elastomer to the client . \nThis depends on the region to which the elastomers are shipped and sold . \n \nCalculation : \nNo specific calculation just apply t ransportation costs invoiced by the manufacturer to the \ncustomers \n \nSource of info: \nManufacturer Logistic \n \nFrequency : \nBased on local Manufacturer Logistic contracts \n \nArchive: \nData will be archived by Finance responsible of activity.', start_char_idx=0, end_char_idx=1065, text_template='[Excerpt from document]\n{metadata_str}\nExcerpt:\n-----\n{content}\n-----\n', metadata_template='{key}: {value}', metadata_seperator='\n'), score=1.0)

Also how and on what basis the score is generated?

[dosubot[bot]]

The scoring mechanism in the query_engine is influenced by several factors and parameters. These include:

1. Temperature: This parameter affects the randomness of the response generation. A higher temperature results in more random responses, while a lower temperature makes the responses more deterministic.

2. Answer Style: This parameter determines the style of the answer generated and can be set to different styles as defined in google.ai.generativelanguage.GenerateAnswerRequest.AnswerStyle.

3. Safety Setting: This parameter is a list of safety settings that can be applied to filter out harmful content.

Additionally, you can customize or adjust the scoring criteria by using node postprocessors like the SimilarityPostprocessor. This allows you to set a threshold on the similarity score, filtering the retrieved nodes based on their similarity to the query. You can also configure query settings directly, such as setting similarity_top_k and applying metadata filters ^[1][2].

If the scoring seems incorrect, you might want to review these parameters and configurations to ensure they align with your expectations. Adjusting the similarity cutoff or applying different filters might help in getting more relevant scores for your nodes.

To continue talking to Dosu, mention @dosu.

[adityamity]

@dosu Below is how i call query engine:

query_engine = RetrieverQueryEngine(
    response_synthesizer =  response_synthesizer, 
    retriever=hybrid_retriever,
)

where can i pass temperature. From where does it take temperature value.?

[dosubot[bot]]

The temperature value can be passed when creating an instance of the OpenAI model, which is then used in the RetrieverQueryEngine. Here's how you can set the temperature:

gpt35turbo_predictor = OpenAI(
    temperature=0,  # Set your desired temperature here
    model_name="gpt-3.5-turbo-0613",
    streaming=True,
    openai_api_key="test-test-test",
)

The temperature value is taken from this OpenAI model instance, which is used to create a ServiceContext. This ServiceContext is associated with the RetrieverQueryEngine through the TreeSelectLeafRetriever, ensuring that the temperature setting is inherited and utilized by the RetrieverQueryEngine ^[1][2].

To continue talking to Dosu, mention @dosu.

[adityamity]

@dosu I am using AzureOpenAI

llm = AzureOpenAI(
    model="gpt-35-turbo",
    deployment_name="gpt-35-turbo",
    api_key=api_key,
    azure_endpoint=azure_endpoint,
    api_version=api_version,
)

here I have not passed the temperature parameter.Also WHat is the default temperature value it is taking.

[dosubot[bot]]

In the AzureOpenAI class, if you don't explicitly set the temperature parameter, it typically defaults to a value of 0.7. This default value is common for many language models, providing a balance between randomness and determinism in the responses. If you want to specify a different temperature, you can add the temperature parameter when initializing the AzureOpenAI instance:

llm = AzureOpenAI(
    model="gpt-35-turbo",
    deployment_name="gpt-35-turbo",
    api_key=api_key,
    azure_endpoint=azure_endpoint,
    api_version=api_version,
    temperature=0.5  # Set your desired temperature here
)

Adjusting the temperature can help you control the variability of the responses generated by the model.

To continue talking to Dosu, mention @dosu.

[adityamity]

@dosu how can i use this same llm to generate response. I am using the below code:

def test_llm(llm, prompt):
    response = llm.generate(prompt)
    return response

prompt = "write a python code to add two numbers"
response = test_llm(llm, prompt)

AttributeError: 'AzureOpenAI' object has no attribute 'generate'

[dosubot[bot]]

To generate a response using the AzureOpenAI object, you should use the complete method instead of generate. Here's how you can modify your code:

def test_llm(llm, prompt):
    response = llm.complete(prompt)
    return response

prompt = "write a python code to add two numbers"
response = test_llm(llm, prompt)
print(response)

The complete method is used for text completion tasks with the AzureOpenAI class ^[1].

To continue talking to Dosu, mention @dosu.

[adityamity]

@dosu I am using below code

llm = AzureOpenAI( model="gpt-35-turbo", deployment_name="gpt-35-turbo", api_key=api_key, azure_endpoint=azure_endpoint, api_version=api_version, temperature=0.2 )

i am getting below error: response = llm.chat.completions.create( 45 model="gpt-4o", 46 messages=[ 47 {"role": "system", "content": system_prompt1 }, 48 ], 49 temperature=0, 50 top_p=0.95, 51 frequency_penalty=0, 52 presence_penalty=0, 53 stop=None 54 ) 56 return response.choices[0].message.content

AttributeError: 'function' object has no attribute 'completions'

[dosubot[bot]]

The error you're encountering is due to incorrect usage of the chat method in the AzureOpenAI class. Instead of using llm.chat.completions.create, you should directly call the chat method on the AzureOpenAI instance. Here's how you can modify your code:

def test_llm(llm, system_prompt1):
    messages = [
        {"role": "system", "content": system_prompt1},
    ]
    response = llm.chat(messages, temperature=0, top_p=0.95, frequency_penalty=0, presence_penalty=0, stop=None)
    return response.message.content

system_prompt1 = "Your system prompt here"
response_content = test_llm(llm, system_prompt1)
print(response_content)

This approach uses the chat method to generate chat completions with the AzureOpenAI class ^[1].

To continue talking to Dosu, mention @dosu.