search

rubra-ai / rubra

Open Weight, tool-calling LLMs
https://docs.rubra.ai/
Apache License 2.0
148 stars 20 forks source link

Incorrect data being fetched from pdf file for some queries. #96

Closed sangee2004 closed 4 months ago

sangee2004 commented 8 months ago

Steps to reproduce the problem:

  1. Create an assistant with custom model and provide a pdf file (travel itinerary) that has details of the plan listed for every day of a 12 day trip.
  2. I am able to get my travel itinerary for some days correctly when I ask question like "What is my Day 9 travel plan ?"

The same query does not work as expected when I ask the same question relating to Day 8 . It actually fetched the details of the trip from Day 10. In fact , the query for Day 10 and Day 8 have the exact same results.

Noticed that the same query results in different behaviors when done multiple times.

I created another assistant with custom model and provided the same pdf file . The same query for Day 8 resulted in fetching trip details of Day 9. In this case , the query for Day 9 and Day 8 have the exact same results.

After couple of hours , I see the same query returning a completely different result

Calling function : FileKnowledge with args: {'query': 'what is my day 8 travel plan'}
2024-02-22 12:53:21 [2024-02-22 20:53:21,364: INFO/ForkPoolWorker-8] HTTP Request: POST http://litellm:8002/v1/chat/completions "HTTP/1.1 200 OK"
2024-02-22 12:53:21 [2024-02-22 20:53:21,365: WARNING/ForkPoolWorker-8] message: The provided search results do not include a day 8 travel plan. However, the last mentioned itinerary is for Day 12.

For comparison , i did the same testing with an assistant with gpt4 model and provided it with the same pdf file (travel itinerary) . In this case the same query returns the correct results.