unable to use graph memory

[Divyanshu9822]

🐛 Describe the bug

I was trying to utilise graph memory using gemini and google embeddings along with qdrant. Pls help

(venv) ~/Desktop/mem0-test-try $ python assistant_v1.py 

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Traceback (most recent call last):
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1280, in completion
    response = client.post(url=url, headers=headers, json=data)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 371, in post
    raise e
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 357, in post
    response.raise_for_status()
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/httpx/_models.py", line 763, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:generateContent?key=AIzaSyCPEtiXJMGiy1tAodWKM0i_klX0uilbusg'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/main.py", line 2152, in completion
    response = vertex_chat_completion.completion(  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1284, in completion
    raise VertexAIError(status_code=error_code, message=err.response.text)
litellm.llms.vertex_ai_and_google_ai_studio.common_utils.VertexAIError: {
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unknown name \"additionalProperties\" at 'tools[0].function_declarations[0].parameters': Cannot find field.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "field": "tools[0].function_declarations[0].parameters",
            "description": "Invalid JSON payload received. Unknown name \"additionalProperties\" at 'tools[0].function_declarations[0].parameters': Cannot find field."
          }
        ]
      }
    ]
  }
}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/users1/Desktop/mem0-test-try/assistant_v1.py", line 67, in <module>
    m.add(memory, user_id="divyanshu", metadata=meta)
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/mem0/memory/main.py", line 110, in add
    graph_result = future2.result()
                   ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/mem0/memory/main.py", line 225, in _add_to_graph
    added_entities = self.graph.add(data, filters)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/mem0/memory/graph_memory.py", line 54, in add
    search_output = self._search(data, filters)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/mem0/memory/graph_memory.py", line 167, in _search
    search_results = self.llm.generate_response(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/mem0/llms/litellm.py", line 85, in generate_response
    response = litellm.completion(**params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/utils.py", line 1074, in wrapper
    raise e
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/utils.py", line 957, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/main.py", line 2920, in completion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2033, in exception_type
    raise e
  File "/Users/users1/Desktop/mem0-test-try/venv/lib/python3.12/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 1049, in exception_type
    raise BadRequestError(
litellm.exceptions.BadRequestError: litellm.BadRequestError: VertexAIException BadRequestError - {
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unknown name \"additionalProperties\" at 'tools[0].function_declarations[0].parameters': Cannot find field.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "field": "tools[0].function_declarations[0].parameters",
            "description": "Invalid JSON payload received. Unknown name \"additionalProperties\" at 'tools[0].function_declarations[0].parameters': Cannot find field."
          }
        ]
      }
    ]
  }
}

ERROR:neo4j.io:Failed to write data to connection ResolvedIPv4Address(('34.124.169.171', 7687)) (ResolvedIPv4Address(('34.124.169.171', 7687)))
ERROR:neo4j.io:Failed to write data to connection IPv4Address(('d12681a6.databases.neo4j.io', 7687)) (ResolvedIPv4Address(('34.124.169.171', 7687)))
(venv) ~/Desktop/mem0-test-try $ 

#requirements.txt
git+https://github.com/Divyanshu9822/mem0.git@main
chromadb
ollama
litellm
neo4j
google-generativeai
vertexai
python-dotenv

#main.py
import os
from mem0 import Memory
from pprint import pprint
from dotenv import load_dotenv

load_dotenv()

GOOGLE_API_KEY = os.getenv("GOOGLE_API_KEY")
neo4j_url = os.getenv("NEO4J_URI")
neo4j_user = os.getenv("NEO4J_USERNAME")
neo4j_password = os.getenv("NEO4J_PASSWORD")
qdrant_url = os.getenv("QDRANT_URL")
qdrant_api_key = os.getenv("QDRANT_API_KEY")

# Configuration for Memory
config = {
    "llm": {
        "provider": "litellm",
        "config": {
            "model": "gemini/gemini-1.5-pro",
            "temperature": 0.2,
            "max_tokens": 1024,
        },
    },
    "embedder": {
        "provider": "gemini",
        "config": {
            "model": "models/text-embedding-004",
            "embedding_dims": 768,
        },
    },
    "graph_store": {
        "provider": "neo4j",
        "config": {
            "url": neo4j_url,
            "username": neo4j_user,
            "password": neo4j_password,
        },
    },
    "vector_store": {
        "provider": "qdrant",
        "config": {
            "collection_name": "test",
            "url": qdrant_url,
            "api_key": qdrant_api_key,
            "embedding_model_dims": 768,
        },
    },
    "version": "v1.1",
}

# Initialize Memory with the configuration
m = Memory.from_config(config)

# Add 5 distinct memories with metadata
memories_to_add = [
    ("I recently started a new job as a software engineer", {"category": "career"}),
    ("I traveled to Japan last year for vacation", {"category": "travel"}),
    ("I bought a new bicycle to stay active", {"category": "fitness"}),
    ("My favorite dish is lasagna", {"category": "food"}),
    ("I have a pet dog named Bruno", {"category": "personal"}),
]

# Add each memory to the memory store with metadata and user_id
for memory, meta in memories_to_add:
    m.add(memory, user_id="divyanshu", metadata=meta)

[prateekchhikara]

Hi @Divyanshu9822 I used your messages and tried creating the graph. It worked for me. I believe there is an issue with your neo4j setup.

[Divyanshu9822]

Hi @Divyanshu9822 I used your messages and tried creating the graph. It worked for me. I believe there is an issue with your neo4j setup.

Did u use Gemini only ? The same config I shared ?

[femto]

Hello, @prateekchhikara facing the same problem, when adding graph mem, it only uses the following two, I can't even use openai.

 if self.llm_provider in ["azure_openai_structured", "openai_structured"]:
            _tools = [
                UPDATE_MEMORY_STRUCT_TOOL_GRAPH,
                ADD_MEMORY_STRUCT_TOOL_GRAPH,
                NOOP_STRUCT_TOOL,
            ]

[Divyanshu9822]

OpenAI is working properly with vector db and graph as well. However it doesn't serve my purpose well, I'm trying to build an AI assistant and less latency is quite important here so i chose gemini flash but it's not working with graph memory there's some issue it graph tools calling, also when i tried to use neo4j remotely it was ever slower so i had to move to local setup. @prateekchhikara can u pls look into the litellm gemini usage in graph memory.

my config

config = {
    "graph_store": {
        "provider": "neo4j",
        "config": {
            "url": NEO4J_URI,
            "username": NEO4J_USERNAME,
            "password": NEO4J_PASSWORD,
        },
    },
    "vector_store": {
        "provider": "qdrant",
        "config": {
            "collection_name": QDRANT_COLLECTION_NAME,
            "url": QDRANT_URL,
            "api_key": QDRANT_API_KEY,
        },
    },
    "version": "v1.1",
}