search

GoogleCloudPlatform / generative-ai

Sample code and notebooks for Generative AI on Google Cloud, with Gemini on Vertex AI
https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview
Apache License 2.0
6.81k stars 1.83k forks source link

[Bug]: Agent almost never generates answer despite data is available in datastore. #913

Open jeromemassot opened 1 month ago

jeromemassot commented 1 month ago

File Name

tutorial_vertex_ai_search_rag_agent.ipynb

What happened?

For all the questions proposed to the deployed agent in this notebook, the agent replies : "I cannot fulfill this request. The available tools do not have information about ..." despite the queried information is present in the structure data.

Relevant log output

No response

Code of Conduct

koverholt commented 1 month ago

Thanks for opening this issue. I attempted to reproduce but was not able to, the notebook is working as expected for me and is returning movies based on records in the dataset.

Ensure that you followed the tutorial steps here: https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#structured-data

Ensure that the search app is working as expected when you preview the search app and search for a sample query: https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#preview_your_app

Ensure that the code cell that tests the Vertex AI Search retriever function is working as expected:

search_kaggle_movies("Harry Potter")

And if you continue to run into errors, can you post more details about:

jeromemassot commented 1 month ago

Hi Kristopher, thanks for your comment. I confirm that the Datastore is consistent with the Kaggle dataset. And if I query the agent 10 times I may have sometimes an answer close to the expected output. However, most of the time, the agent is not capable to generate the answer despite the retrieved context seems ok (tested directly in the Agent Builder). So I confirm that the overall performance of the agent is very low in my environment despite a consistent datastore. Best Regards Jerome

PS: the remote agent has same issue. Deployed using the given code in the notebook

remote_agent = reasoning_engines.ReasoningEngine.create(
    agent,
    requirements=[
        "google-cloud-aiplatform[langchain,reasoningengine]",
        "cloudpickle==3.0.0",
        "pydantic==2.7.4",
        "langchain-google-community",
        "google-cloud-discoveryengine",
    ],
)
koverholt commented 1 month ago

Thanks for the info that the local agent is having the same issue. In order for me to reproduce, can you post more details about: