Issue: Adjusting Model Temperature

[B14ckFox]

Description I am using the following example code with LightRAG and the gpt_4o_mini_complete model. However, I noticed that the model generates made-up information that doesn't align with the provided knowledge graph. I would like the model to strictly adhere to the provided context and avoid introducing hallucinations.

Here are my specific questions:

How can I adjust the temperature of the model in this example to make the responses more deterministic and context-based? Should I also adapt the system prompt to enforce stricter adherence to the provided graph? If so, could you provide guidance or an example of how to adjust the prompt effectively?

testing after kg creation: --> not accurate response too much made up

from lightrag import LightRAG, QueryParam
from lightrag.llm import gpt_4o_mini_complete

# working directory
WORKING_DIR = "./custom_kg"

rag = LightRAG(
    working_dir=WORKING_DIR,
    llm_model_func=gpt_4o_mini_complete, 
)

# Query the existing knowledge graph
question1 = "What products are developed by CompanyA?"
response1 = rag.query(question1, param=QueryParam(only_need_context=True, mode="local")) 
print(f"Question: {question1}\nResponse: {response1}\n")

question2 = "Where is EventY hosted?"
response2 = rag.query(question2, param=QueryParam(mode="local")) 
print(f"Question: {question2}\nResponse: {response2}")

Code from inster_custom_kg.py

import os
from lightrag import LightRAG, QueryParam
from lightrag.llm import gpt_4o_mini_complete

WORKING_DIR = "./custom_kg"

if not os.path.exists(WORKING_DIR):
    os.mkdir(WORKING_DIR)

rag = LightRAG(
    working_dir=WORKING_DIR,
    llm_model_func=gpt_4o_mini_complete,  # Use gpt_4o_mini_complete LLM model
    # llm_model_func=gpt_4o_complete  # Optionally, use a stronger model
)

custom_kg = {
    "entities": [
        {
            "entity_name": "CompanyA",
            "entity_type": "Organization",
            "description": "A major technology company",
            "source_id": "Source1"
        },
        {
            "entity_name": "ProductX",
            "entity_type": "Product",
            "description": "A popular product developed by CompanyA",
            "source_id": "Source1"
        },
        {
            "entity_name": "PersonA",
            "entity_type": "Person",
            "description": "A renowned researcher in AI",
            "source_id": "Source2"
        },
        {
            "entity_name": "UniversityB",
            "entity_type": "Organization",
            "description": "A leading university specializing in technology and sciences",
            "source_id": "Source2"
        },
        {
            "entity_name": "CityC",
            "entity_type": "Location",
            "description": "A large metropolitan city known for its culture and economy",
            "source_id": "Source3"
        },
        {
            "entity_name": "EventY",
            "entity_type": "Event",
            "description": "An annual technology conference held in CityC",
            "source_id": "Source3"
        },
        {
            "entity_name": "CompanyD",
            "entity_type": "Organization",
            "description": "A financial services company specializing in insurance",
            "source_id": "Source4"
        },
        {
            "entity_name": "ServiceZ",
            "entity_type": "Service",
            "description": "An insurance product offered by CompanyD",
            "source_id": "Source4"
        }
    ],
    "relationships": [
        {
            "src_id": "CompanyA",
            "tgt_id": "ProductX",
            "description": "CompanyA develops ProductX",
            "keywords": "develop, produce",
            "weight": 1.0,
            "source_id": "Source1"
        },
        {
            "src_id": "PersonA",
            "tgt_id": "UniversityB",
            "description": "PersonA works at UniversityB",
            "keywords": "employment, affiliation",
            "weight": 0.9,
            "source_id": "Source2"
        },
        {
            "src_id": "CityC",
            "tgt_id": "EventY",
            "description": "EventY is hosted in CityC",
            "keywords": "host, location",
            "weight": 0.8,
            "source_id": "Source3"
        },
        {
            "src_id": "CompanyD",
            "tgt_id": "ServiceZ",
            "description": "CompanyD provides ServiceZ",
            "keywords": "provide, offer",
            "weight": 1.0,
            "source_id": "Source4"
        }
    ]
}

rag.insert_custom_kg(custom_kg)

The model should:

• Answer strictly based on the context provided in the knowledge graph.
• Avoid generating additional, fabricated information. Questions

Temperature Adjustment: Where and how can I configure the model's temperature in the above example? Prompt Adaptation: Would modifying the system prompt improve context adherence, and if so, how should I approach this?

Thanks in advance for the help and guidance!

[cypridox]

I recently started testing LightRAG as well (thank you for developing this promising repository). I have encountered the same observation as the author of this issue: the model is insufficiently strict in adhering to the provided context and often produces generic answers.

While adjusting the temperature setting is a potential solution, my usual approach would be to resolve this issue through the system prompt. I attempted to add an instruction directly to the question (e.g., "limit your answer to the provided context"), but this resulted in no answers being generated, as the model deemed nothing to be in context.

I would, therefore, like to have control over modifying the system prompt and the overall query prompt. I reviewed the Python code to locate where these prompts are defined or managed. Unfortunately, the (undocumented) code appears somewhat inaccessible in this regard. For instance, prompt.py seems primarily intended for generating the knowledge graph.

Where in the codebase can I view and customize the system prompt and the query prompt used for generating answers from the model?

[LarFii]

The prompt used for generating responses is defined in the prompt.py file, and it is called rag_response.

[B14ckFox]

and this is how I adjust the temperature?

async def gpt_4o_mini_complete( prompt, system_prompt=None, history_messages=[], kwargs ) -> str: kwargs.setdefault("temperature", 0) return await openai_complete_if_cache( "gpt-4o-mini", prompt, system_prompt=system_prompt, history_messages=history_messages, kwargs, )

[sebastianschramm]

@B14ckFox I believe you can also directly pass your llm parameters via llm_model_kwargs to LightRAG without modifying anything else:

rag = LightRAG(
    working_dir=WORKING_DIR,
    llm_model_func=gpt_4o_mini_complete,
    llm_model_kwargs={"temperature": 0.0}
)