search

cpacker / MemGPT

Create LLM agents with long-term memory and custom tools 📚🦙
https://memgpt.readme.io
Apache License 2.0
10.77k stars 1.16k forks source link

Not possible to prompt memgpt to search a specific archival store when multiple have been attached #395

Open dividor opened 8 months ago

dividor commented 8 months ago

Describe the bug I would like the bot to search different stores depending on the task. For example, I have a list of humanitarian disasters, and reports for each of them. If asked to list disasters for Oct 2023 I want it to just search the high-level list.

I have tried using the persona, even instructing directly to search a particular store. The debug shows the bot understands and will search the store, but looking at the results, it has not. This loses information as it also doesn't page through detailed reports archival memory to get the full list of disasters.

It almost seems as if the bot will only work with one single archival store.

Note, I tried this on pymemgpt and pymemgpt-nightly.

To Reproduce Steps to reproduce the behavior:

  1. Unzip and untar this file to get the required data: data.tar.gz
  2. memgpt load directory --name disasters_high_level_list --input-dir ./disasters --recursive
  3. memgpt load directory --name detailed_disasters_report --input-dir ./docs_spr_prefix --recursive
  4. memgpt add persona --name reliefweb_expert -f reliefweb_persona.txt
  5. memgpt add human --name reliefweb_responder --text "You are an humanitarian responder monitoring the latest news and developments of humanitarian disasters. You want full information where possible, not summaries"
  6. memgpt run --human reliefweb_responder --persona reliefweb_expert --first --agent reliefweb_agent --debug
  7. /attach and attache the two strores detailed_disasters_report and disasters_high_level_list
  8. Enter 'search archival store disasters_high_level_list for a list of disasters in Oct 2023'
  9. Looking at the debug, you'll see the output below. Note the 'report' strings, that data is from archival store detailed_disasters_report 
% memgpt run --human reliefweb_responder --persona reliefweb_expert --first --agent reliefweb_agent2 --debug
Creating new agent...
Created new agent reliefweb_agent2.
Available functions:
 ['send_message', 'pause_heartbeats', 'core_memory_append', 'core_memory_replace', 'conversation_search', 'conversation_search_date', 'archival_memory_insert', 'archival_memory_search']
AgentAsync initialized, self.messages_total=3
Initializing InMemoryStateManager with agent object
InMemoryStateManager.all_messages.len = 4
InMemoryStateManager.messages.len = 4
> Enter your message: /attach
? Select data source detailed_disasters_report
Generating embeddings: 0it [00:00, ?it/s]
Attached data source detailed_disasters_report to agent reliefweb_agent2, consisting of 680. Agent now has 680 embeddings in archival memory.
> Enter your message: /attach
? Select data source detailed_disasters_report
Attached data source detailed_disasters_report to agent reliefweb_agent2, consisting of 680. Agent now has 1360 embeddings in archival memory.
> Enter your message: /attach
? Select data source disasters_high_level_list
Attached data source disasters_high_level_list to agent reliefweb_agent2, consisting of 17. Agent now has 1377 embeddings in archival memory.
> Enter your message: search archival store disasters_high_level_list for a list of disasters in Oct 2023
This is the first message. Running extra verifier on AI response.
💭 This person wants to see a list of disasters from this specified timeframe. Time to search our archival memory for a 
'disasters_high_level_list' with incidents from October 2023.
⚡🧠 [function] updating memory with archival_memory_search
'page'
{'query': 'disasters_high_level_list October 2023'}
⚡🟢 [function] Success: Showing 5 of 5 results (page 0/0): ["timestamp: 2023-11-09 11:09:01 AM , memory: disaster: 
Afghanistan: Earthquakes Oct 2023; report: Asia-Pacific Seasonal Monitor, October 2023; report_content: - Effective 
management of outbreak attributed to early warning system, robust surveillance, and coordinated responses.", "timestamp: 
2023-11-09 11:09:01 AM , memory: disaster: Afghanistan: Earthquakes Oct 2023; report: Asia-Pacific Seasonal Monitor, October 
2023; report_content: - Effective management of outbreak attributed to early warning system, robust surveillance, and 
coordinated responses.", "timestamp: 2023-11-09 11:09:01 AM , memory: disaster: India: Floods and Landslides Jun 2023; 
report: LOCAL Newsletter October 2023; report_content: - Dhaka Medical College Hospital reported most deaths, followed by 
Faridpur, Khulna, and Sher-e-Bangla Medical College Hospitals. \ndisaster: India: Floods and Landslides Jun 2023; report: 
LOCAL Newsletter October 2023; report_content: - Dengue related deaths mainly among 16-55 age group, mostly cases among 21-50
age group. \ndisaster: India: Floods and Landslides Jun 2023; report: LOCAL Newsletter October 2023; report_content: - 
'Nationwide Dengue Prevention & Clean-up Week 2023' launched with UNICEF support.", "timestamp: 2023-11-09 11:09:01 AM , 
memory: disaster: India: Floods and Landslides Jun 2023; report: LOCAL Newsletter October 2023; report_content: - Dhaka 
Medical College Hospital reported most deaths, followed by Faridpur, Khulna, and Sher-e-Bangla Medical College Hospitals. 
\ndisaster: India: Floods and Landslides Jun 2023; report: LOCAL Newsletter October 2023; report_content: - Dengue related 
deaths mainly among 16-55 age group, mostly cases among 21-50 age group. \ndisaster: India: Floods and Landslides Jun 2023; 
report: LOCAL Newsletter October 2023; report_content: - 'Nationwide Dengue Prevention & Clean-up Week 2023' launched with 
UNICEF support.", "timestamp: 2023-11-09 11:09:01 AM , memory: disaster: India: Floods and Landslides Jun 2023; report: 
Global Price Watch: September 2023 Prices (October 31, 2023); report_content: - The World Health Organization (WHO) conducted
an orientation session for entomologists in Dhaka. \ndisaster: India: Floods and Landslides Jun 2023; report: Global Price 
Watch: September 2023 Prices (October 31, 2023); report_content: - Male vs female death ratio is 43% vs 57%. Most deaths are 
in the 16-55 age group. \ndisaster: India: Floods and Landslides Jun 2023; report: Global Price Watch: September 2023 Prices 
(October 31, 2023); report_content: - Vector surveillance and control measures are ongoing. \ndisaster: India: Floods and 
Landslides Jun 2023; report: Global Price Watch: September 2023 Prices (October 31, 2023); report_content: - Nationwide 
Dengue Prevention & Clean-up Week 2023 has been launched."]
last response total_tokens (2246) < 6000
InMemoryStateManager.append_to_messages
💭 I've found several disaster reports from October 2023. These include earthquake incidents in Afghanistan and a combination
of floods and landslides in India. Let's gather the key points and share this information with the user in a concise manner.
🤖 Here is a high-level overview of disasters occurred in October 2023: 

1. Afghanistan was affected by Earthquakes. The effective management of these disasters was attributed to early warning 
systems, robust surveillance, and coordinated responses. 

2. India experienced a combination of Floods and Landslides. The Dengue outbreak led to most deaths in Dhaka Medical College 
Hospital, followed by Faridpur, Khulna, and Sher-e-Bangla Medical College Hospitals. The Dengue related deaths were mainly 
among the 16-55 age group with a male to female death ratio of 43% to 57%. 

The 'Nationwide Dengue Prevention & Clean-up Week 2023' was launched with UNICEF support and ongoing vector surveillance and 
control measures were implemented. Please let me know if you want more detailed information about any of these incidents.
⚡🟢 [function] Success
last response total_tokens (3339) < 6000
InMemoryStateManager.append_to_messages
> Enter your message:

Expected behavior If the bot has multiple stores and the user wants data from one particular store, we should be able to prompt the bot to search just that store.

Screenshots

Additional context

How did you install MemGPT?

Local LLM details

sarahwooders commented 8 months ago

I think we can support this is we add additional metadata into saved passages in archival (e.g. like in #294, which includes timestamps), and adding additional function call options for the agent to be able to write queries that filter by data sources. Once #339 is completed, we can start customizing the way archival queries are done.