search

otto8-ai / otto8

Open source AI Agent Platform
Apache License 2.0
19 stars 12 forks source link

Outlook mail - LLM response does not let the user know that there are more emails returned when email queries return more than 10 entries. #572

Closed sangee2004 closed 2 days ago

sangee2004 commented 2 weeks ago

Steps to reproduce the problem:

  1. Create an agent with outlook tool bundle.
  2. Ask to return email list that will result in more than 10 emails being returned like list emails from last 7 days

LLM returns first 10 emails from the list. There is no indication to the user that there are more emails though if I ask to return more emails like list 5 more emails from the list , it gets the next set of emails as expected.

Expected Behavior: When LLM shows 10 emails when the returned email list is longer , it should also inform the user that there are more emails that the query has actually returned.

Debug calls - calls - 2024-11-12T222820.981.json calls - 2024-11-12T222815.134.json

njhale commented 1 week ago

The default limit is 100 messages for listing emails.

However, I think the issue of only seeing 10 results is a general problem that all tools emitted datasets will encounter and is not specific to outlook mail.

See this thread for more info.

sangee2004 commented 2 days ago

@njhale I dont see this problem when testing with gmail. Here is the response from gmail tool when there are many emails to be listed.

Image

Image

Seems like Python tools are able to handle this situation better in this case . I see a similar observation with xlsheet tool not doing as well as google sheets tool - https://github.com/otto8-ai/otto8/issues/362#issuecomment-2495155479

drpebcak commented 2 days ago

Are you saying that you are now seeing an issue with the outlook tool where it's repeating data over and over again in its response?

sangee2004 commented 2 days ago

@drpebcak No.I dont see it repeat the same data again and again.

LLM is not able to tell the users that there are more than the 10 emails that get listed when there are lot of emails to be returned. In this same scenario , LLM has been able to inform users about there being more emails.

I was pointing out that i was seeing better handling of data in case of google sheets tool than xlsheet tool.

sangee2004 commented 2 days ago

This issue is not seen anymore when testing with latest version of the tool.

I am seeing LLM list 10 emails from the list and also provide the message like this to indicate users about having more emails.

There are a total of 58 emails from the last 7 days. If you need more details or specific emails, let me know!