Context is not passed into the rails.generate function

[jackchan0528]

Following the doc here: https://github.com/NVIDIA/NeMo-Guardrails/tree/develop/docs/getting_started/7_rag

response = rails.generate(messages=[{
    "role": "context",
    "content": {
        "relevant_chunks": """
            Employees are eligible for the following time off:
              * Vacation: 20 days per year, accrued monthly.
              * Sick leave: 15 days per year, accrued monthly.
              * Personal days: 5 days per year, accrued monthly.
              * Paid holidays: New Year's Day, Memorial Day, Independence Day, Thanksgiving Day, Christmas Day.
              * Bereavement leave: 3 days paid leave for immediate family members, 1 day for non-immediate family members. """
    }
},{
    "role": "user",
    "content": "How many vacation days do I have per year?"
}])
print(response["content"])

The correct answer should be 20 days per year. However the model hallucinated and answered:

You are entitled to 15 vacation days per year based on your current employment contract. These days accrue gradually throughout the year, so you may not have all 15 days available right at the beginning of the year. It's always a good idea to check with your HR department or refer to your employment agreement for specific details on how vacation days are accrued and how they can be used. If you have any specific plans for using your vacation days, I can help you keep track of them and even suggest some travel destinations or activities for your time off. Just let me know how I can assist you further!

I tried lots of different context and question pairs. It seems that the relevant_chunks was not passed into the entire prompt for generating the answer.

[drazvan]

@jackchan0528 : did you test with the provided notebook? (I've just tried it again and it works correctly)

[jackchan0528]

@drazvan I rerun the notebook and still get the wrong answer as 15 days per year, while the answer should be 20. You may try even simpler test cases and see that the answer generated is irrelevant to the context provided.

response = rails.generate(messages=[{
    "role": "context",
    "content": {
        "relevant_chunks": """
            Employees are eligible for the following time off:
              * Vacation: 20 days per year, accrued monthly.
              * Sick leave: 15 days per year, accrued monthly.
              * Personal days: 5 days per year, accrued monthly.
              * Paid holidays: New Year's Day, Memorial Day, Independence Day, Thanksgiving Day, Christmas Day.
              * Bereavement leave: 3 days paid leave for immediate family members, 1 day for non-immediate family members. """
    }
},{
    "role": "user",
    "content": "How many vacation days do I have per year?"
}])
print(response["content"])

Output:

You have 15 vacation days per year. This includes both paid time off and any additional days you may have accrued based on your years of service with the company. If you need to take more days off than what is provided, you can always discuss options with your supervisor or HR department to see if any additional time can be arranged. Is there anything else I can help you with?

nemoguardrails version: 0.8.1 config/ file contains only config.yml with content below (I tried having and removing the input & output rails for this testing):

models:
  - type: main
    engine: openai
    model: gpt-3.5-turbo-0125

[drazvan]

@jackchan0528 : this is most likely because you've changed the model to gpt-3.5-turbo-0125 (from gpt-3.5-turbo-instruct). Can you confirm that use the instruct version works correctly on your end? cc @trebedea

[jackchan0528]

@drazvan I tried swapping the model to gpt-3.5-turbo-instruct, by trying 5 times to generate the answers, but the model hallucinated even worse. May I understand if this never happened from your end?

[drazvan]

On my end, running the notebook as-is yields the correct result:

[jackchan0528]

Hi @drazvan , I figured out that the context is correctly read in ONLY when the rails/ folder exist. I used to have only config.yml and prompts.yaml. After reviewing the setting at the documentation, which has actions.py and rails/ which contains blocked_terms.co and disallowed_topics.co, I added them back and the context is read in successfully and the result is correct. I have done further checking, and the result won't work if you delete any of the 2 .co files under rails. The result simply answer wrongly without any errors. (meaning that if you delete blocked_terms.co OR disallowed_topics.co alone, the context will not be read in properly. For the reasons of it I am still investigating on, because logically it should not affect how the context is read in.

[jackchan0528]

Update: we don't even need actions.py nor blocked_terms.co. We simply need content in rails/disallowed_topics.co to yield a correct answer based on the context given