search

dust-tt / dust

Amplify your team's potential with customizable and secure AI assistants.
https://dust.tt
MIT License
957 stars 108 forks source link

Check that semantic search retrieval with auto timeframe performs well enough #1926

Open philipperolet opened 1 year ago

philipperolet commented 1 year ago
philipperolet commented 1 year ago

Tested with bots @slack and @dust. Both bots always got the same timeframe, as ~expected

👍 When was aric recruited? => All time ~ Were there any major incidents in production last month? => during the last month (meaning last 30 days rather than most likely expected September) ~ What is Daphné currently working on? => All time x~ Did we discuss the onboarding process yesterday? => During the last day (so if I ask at 5pm, i'll probably get nothing) ~ Link to the last roadmap document for 2023? => All time ~ What were last week's main goals for the team? => during the last week ~ What are this week's main goals for the team? => during the last 7 days x List all the goals of Stanislas since last tuesday / of Daphné since last wednesday => all time 👍 What is the command line to upgrade a workspace? => all time x How many incidents did we have since March? => all time x What happened on october 1st? => all time

spolu commented 1 year ago

@philipperolet one quick fix we can do for this is the following:

We do inject some "context" for the generator but we don't for the input-generator.

You can see the differences here: generator: https://dust.tt/w/78bda07b39/a/6a27050429/runs/78487bc30490891b7b8527703617daa91f176470bfaa394d5b80ed43d956f449 inputs-generator: https://dust.tt/w/78bda07b39/a/f4816b1e13/runs/6d3522f0010feea57f6fb0e91661c828d37fcd5df2e418a6e6ef625940e09d0e

(looking at the prompt passed to both)

This prevent the inputs generator from answer any question that is relative to the day today (last tuesday, october 1, etc...)

(Sorry again for the reaction earlier)

spolu commented 1 year ago

Generation prompt construction: https://dust.tt/w/78bda07b39/a/6a27050429/runs/78487bc30490891b7b8527703617daa91f176470bfaa394d5b80ed43d956f449

Inputs generator prompt construction: https://github.com/dust-tt/dust/blob/main/front/lib/api/assistant/agent.ts#L43

Sharing the same prompt construction should solve a number of the queries above.

spolu commented 1 year ago

For the context the goal was also to test this in the context of injecting the timeframe as part of the instruction and check that the model picks it up