search

opensearch-project / ml-commons

ml-commons provides a set of common machine learning algorithms, e.g. k-means, or linear regression, to help developers build ML related features within OpenSearch.
Apache License 2.0
83 stars 118 forks source link

Unable to execute the Agent flow. #2482

Open admin-techheralds opened 1 month ago

admin-techheralds commented 1 month ago

Describe the bug

Hi, I have setup the opensearch cluster to run on single instance. I have created the embedding model and setup the pipeline to do the text-embedding on indexing the documents. I am able to query the index to get the records. To setup RAG, I have created the remote connector and able to create the model and deployed it. Using the VectorDBTool and MLModelTool, I have created the agent. On executing the error, I am observing the following error: "{\"error\":{\"reason\":\"Invalid Request\",\"details\":\"Error from remote service: \",\"type\":\"OpenSearchStatusException\"},\"status\":400}"

All the above, I have tried it in DevTools of the dashboard web.

Related component

Plugins

To Reproduce

Below is the list of configurations I have enabled:

Cluster settings:

_GET /cluster/settings

{
  "persistent": {
    "plugins": {
      "ml_commons": {
        "rag_pipeline_feature_enabled": "true",
        "agent_framework_enabled": "true",
        "memory_feature_enabled": "true",
        "only_run_on_ml_node": "false",
        "trusted_connector_endpoints_regex": [
          """^https://runtime\.sagemaker\..*[a-z0-9-]\.amazonaws\.com/.*$""",
          """^https://api\.openai\.com/.*$""",
          """^https://api\.cohere\.ai/.*$""",
        ],
        "model_access_control_enabled": "true",
        "native_memory_threshold": "99",
        "allow_registering_model_via_local_file": "true",
        "connector_access_control_enabled": "true",
        "allow_registering_model_via_url": "true"
      },
      "index_state_management": {
        "template_migration": {
          "control": "-1"
        }
      }
    }
  },
  "transient": {}
}

Embedding model settings:

_GET /_plugins/_ml/models/cyEKpY8BQusrTeCRBnv

{
  "name": "huggingface/sentence-transformers/all-MiniLM-L12-v2",
  "model_group_id": "4BQThY8B2f3NeJALZupD",
  "algorithm": "TEXT_EMBEDDING",
  "model_version": "7",
  "model_format": "TORCH_SCRIPT",
  "model_state": "DEPLOYED",
  "model_content_size_in_bytes": 134568911,
  "model_content_hash_value": "f8012a4e6b5da1f556221a12160d080157039f077ab85a5f6b467a47247aad49",
  "model_config": {
    "model_type": "bert",
    "embedding_dimension": 384,
    "framework_type": "SENTENCE_TRANSFORMERS",
    "all_config": """{"_name_or_path":"microsoft/MiniLM-L12-H384-uncased","attention_probs_dropout_prob":0.1,"gradient_checkpointing":false,"hidden_act":"gelu","hidden_dropout_prob":0.1,"hidden_size":384,"initializer_range":0.02,"intermediate_size":1536,"layer_norm_eps":1e-12,"max_position_embeddings":512,"model_type":"bert","num_attention_heads":12,"num_hidden_layers":12,"pad_token_id":0,"position_embedding_type":"absolute","transformers_version":"4.8.2","type_vocab_size":2,"use_cache":true,"vocab_size":30522}"""
  },
  "created_time": 1716460864731,
  "last_updated_time": 1716797846671,
  "last_registered_time": 1716460884584,
  "last_deployed_time": 1716797846671,
  "auto_redeploy_retry_times": 0,
  "total_chunks": 14,
  "planning_worker_node_count": 1,
  "current_worker_node_count": 1,
  "planning_worker_nodes": [
    "SgpCQj20SGWdpTIL0-bXGg"
  ],
  "deploy_to_all_nodes": true,
  "is_hidden": false
}

Index pipeline settings:

_GET /_ingest/pipeline/song_lyricspipeline

{
  "song_lyrics_pipeline": {
    "description": "Song Lyrics pipeline",
    "processors": [
      {
        "text_embedding": {
          "model_id": "cyEKpY8BQusrTe_CRBnv",
          "field_map": {
            "lyrics": "vector_lyrics"
          }
        }
      }
    ]
  }
}

Index settings:

_GET /song_lyrics_index/settings

{
  "song_lyrics_index": {
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "number_of_shards": "1",
        "provided_name": "song_lyrics_index",
        "knn.space_type": "cosinesimil",
        "default_pipeline": "song_lyrics_pipeline",
        "knn": "true",
        "creation_date": "1716470171990",
        "number_of_replicas": "1",
        "uuid": "hFv5slJlRr-9DVkYCxSH1A",
        "version": {
          "created": "136347827"
        }
      }
    }
  }
}

Index mappings:

_GET /song_lyrics_index/mapping

{
  "song_lyrics_index": {
    "mappings": {
      "properties": {
        "album": {
          "type": "text"
        },
        "artist": {
          "type": "text"
        },
        "id": {
          "type": "text"
        },
        "lyrics": {
          "type": "text"
        },
        "meta_data": {
          "properties": {
            "media": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "release_date": {
              "type": "date"
            },
            "url": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "writers": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "year": {
              "type": "float"
            }
          }
        },
        "vector_lyrics": {
          "type": "knn_vector",
          "dimension": 384
        }
      }
    }
  }
}

Sample Index data - vector_lyrics field - purposely removed that data as it is huge (embedding field)...:

{
    "_index": "song_lyrics_index",
    "_id": "1caeb10095ab4c419eae32d6dab2cb0e",
    "_score": 1,
    "_source": {
      "artist": "Connie Francis",
      "album": "Rock ‘n’ Roll Million Sellers",
      "meta_data": {
        "release_date": "1959-01-01",
        "year": 1959,
        "media": "[{'provider': 'youtube', 'start': 0, 'type': 'video', 'url': 'http://www.youtube.com/watch?v=V8x5cUFoDnU'}]",
        "writers": "[]",
        "url": "https://genius.com/Connie-francis-lipstick-on-your-collar-lyrics"
      },
      "lyrics": "When you left me all alone At the record hop Told me you were going out For a soda pop You were gone for quite awhile Half an hour or more You came back And man, oh man This is what I saw  Chorus Lipstick on your collar Told a tale on you Lipstick on your collar Said you were untrue Bet your bottom dollar You and I are through  Cause lipstick on your collar Told a tale on you, yeah [Instrumental Interlude] You said it belonged to me Made me stop and think And then I noticed yours was red Mine was baby pink Who walked in but Mary Jane Lipstick all a mess Were you smoochin' my best friend? Guess the answer's yes  Repeat Chorus  Cause lipstick on your collar Told a tale on you, boy Told a tale on you, man Told a tale on you, yeah",
      "vector_lyrics": [...]
    }
  },
  {
    "_index": "song_lyrics_index",
    "_id": "d10f9cf494bc4b688016929682598fc6",
    "_score": 1,
    "_source": {
      "artist": "Drifters",
      "album": "Under the Boardwalk",
      "meta_data": {
        "release_date": "1959-04-24",
        "year": 1959,
        "media": "[{'provider': 'youtube', 'start': 0, 'type': 'video', 'url': 'http://www.youtube.com/watch?v=noFS4_oEakI'}]",
        "writers": "[{'api_path': '/artists/19688', 'header_image_url': 'https://images.genius.com/469flyng83mlexo1nrpbo5qsn.500x418x1.jpg', 'id': 19688, 'image_url': 'https://images.genius.com/469flyng83mlexo1nrpbo5qsn.500x418x1.jpg', 'is_meme_verified': False, 'is_verified': False, 'name': 'Ben E. King', 'url': 'https://genius.com/artists/Ben-e-king'}, {'api_path': '/artists/41574', 'header_image_url': 'https://images.genius.com/f06bc78c0be8d48120eeecc8318fd359.204x264x1.jpg', 'id': 41574, 'image_url': 'https://images.genius.com/f06bc78c0be8d48120eeecc8318fd359.204x264x1.jpg', 'is_meme_verified': False, 'is_verified': False, 'name': 'Mike Stoller', 'url': 'https://genius.com/artists/Mike-stoller'}, {'api_path': '/artists/41573', 'header_image_url': 'https://images.genius.com/36315105955fb3074737590940b80fb5.300x272x1.jpg', 'id': 41573, 'image_url': 'https://images.genius.com/36315105955fb3074737590940b80fb5.300x272x1.jpg', 'is_meme_verified': False, 'is_verified': False, 'name': 'Jerry Leiber', 'url': 'https://genius.com/artists/Jerry-leiber'}, {'api_path': '/artists/557305', 'header_image_url': 'https://assets.genius.com/images/default_avatar_300.png?1582151316', 'id': 557305, 'image_url': 'https://assets.genius.com/images/default_avatar_300.png?1582151316', 'is_meme_verified': False, 'is_verified': False, 'name': 'George Treadwell', 'url': 'https://genius.com/artists/George-treadwell'}, {'api_path': '/artists/557304', 'header_image_url': 'https://assets.genius.com/images/default_avatar_300.png?1582151316', 'id': 557304, 'image_url': 'https://assets.genius.com/images/default_avatar_300.png?1582151316', 'is_meme_verified': False, 'is_verified': False, 'name': 'Lover Patterson', 'url': 'https://genius.com/artists/Lover-patterson'}]",
        "url": "https://genius.com/The-drifters-there-goes-my-baby-lyrics"
      },
      "lyrics": "[Intro] (Bo-bo) (doo-doot-doo-doo-doo-doo) (There she goes) (doo-doot-doo-doo-doo-doo) (There she goes) (doo-doot-doo-doo-doo-doo) (Bo-bo) (doo-doot-doo-doo) (Bo-bo) (doo-doo-doo-doo)  There goes my baby Movin' on down the line Wonder where, wonder where Wonder where she is bound?  I broke her heart And made her cry Now I'm alone, so all alone What can I do, what can I do?  (There goes my baby) Whoa-oh-oh-oh-oh (There goes my baby) Yeah, yeah, yeah, yeah (There goes my baby) Whoa-oh-oh-oh (There she goes) Yeah! (There she goes)  I wanna know if she loved me Did she really love me? Was she just playing Me for a fool?  I wonder why she left me Why did she leave me So all alone So all alone?  I was gonna tell her that I loved her And that I need her Beside my side To be my guide  I wanna know where is my (doo-doot-doo-doo-doo-doo) Where is my baby? (doo-doot-doo-doo-doo-doo) I want my baby (doo-doot-doo-doo-doo-doo) I need my baby  Yeah, whoa-oh-oh (There goes my baby) Whoa-oh-oh-oh-oh (There goes my baby) Whoa-oh-oh-oh-oh There goes my baby) Whoa-oh-oh-oh-oh (There goes my baby) Whoa-oh-oh-oh-oh",
      "vector_lyrics": [...]
    }
  },
  {
    "_index": "song_lyrics_index",
    "_id": "6c6eb041cf4f4ce7b19207a6ffc452b1",
    "_score": 1,
    "_source": {
      "artist": "Elvis Presley",
      "album": "50,000,000 Elvis Fans Can’t Be Wrong (Elvis’ Gold Records, Vol. 2)",
      "meta_data": {
        "release_date": "1959-06-23",
        "year": 1959,
        "media": "[{'provider': 'youtube', 'start': 0, 'type': 'video', 'url': 'http://www.youtube.com/watch?v=HnrZ7MQsT0s'}]",
        "writers": "[{'api_path': '/artists/1014527', 'header_image_url': 'https://images.genius.com/d2fcd79abf3d5d805bb61d555cb141a4.200x200x1.jpg', 'id': 1014527, 'image_url': 'https://images.genius.com/d2fcd79abf3d5d805bb61d555cb141a4.200x200x1.jpg', 'is_meme_verified': False, 'is_verified': False, 'name': 'Sidney Wyche', 'url': 'https://genius.com/artists/Sidney-wyche'}, {'api_path': '/artists/368580', 'header_image_url': 'https://images.genius.com/33fad1faa4fef3a4605899e42fa642fc.220x220x1.jpg', 'id': 368580, 'image_url': 'https://images.genius.com/33fad1faa4fef3a4605899e42fa642fc.220x220x1.jpg', 'is_meme_verified': False, 'is_verified': False, 'name': 'Aaron Schroeder', 'url': 'https://genius.com/artists/Aaron-schroeder'}]",
        "url": "https://genius.com/Elvis-presley-a-big-hunk-o-love-lyrics"
      },
      "lyrics": "[Intro] Hey baby, I ain't asking much of you No no no no no no no no baby, I ain't asking much of you Just a big-a big-a big-a hunk of love will do  [Verse 1] Don't be a stingy little mama You're about to starve me half to death Well you can spare a kiss or two and Still have plenty left, no no no Baby, I ain't asking much of you Just a big-a big-a big-a hunk of love will do  [Verse 2] You're just a natural born beehive Filled with honey to the top Well I ain't greedy baby All I want is all you got, no no no I ain't asking much of you Just a big-a big-a big-a hunk of love will do  [Verse 3] I got wishbone in my pocket I got a rabbit's foot around my wrist You know I'd have all the things these lucky charms could bring If you'd give me just one sweet kiss, no no no no no no no Baby, I ain't asking much of you Just a big-a big-a big-a hunk of love will do",
      "vector_lyrics": [ ... ]
    }
  },
  {
    "_index": "song_lyrics_index",
    "_id": "6555a24228c74579ac9fe7c6737adb54",
    "_score": 1,
    "_source": {
      "artist": "Johnny and The Hurricanes",
      "album": "Red River Rock",
      "meta_data": {
        "release_date": null,
        "year": 1959,
        "media": "[]",
        "writers": "[]",
        "url": "https://genius.com/Johnny-and-the-hurricanes-red-river-rock-lyrics"
      },
      "lyrics": """Unknown Miscellaneous Paddy And The Whale PADDY AND THE WHALE  Paddy O'Brian left Ireland in glee; He had a strong notion old England to see; He shipped in the Nellie, for England was bound And the whiskey he drank made his head go around  Cho: Laddy whack, fol de dol, fol de rol I dee dee *  O, Paddy been never sailing before; It made his heart ache when he heard the loud roar; For the glance of his eye, a whale he did spy: "I'm going to be ate," says Paddy,"by-and-by"  O, Paddy run forward and caught hold of the mast He grasped his arms round and there he held fast The boat gave a tip, and, losing his grip Down in the whale's belly poor Paddy did slip  He was down in the whale six months and five days Till luck one day to his throat he did pop The whale give a snort and then give a blow And out on the land poor Paddy did go  O, Paddy is landed and safe on the shore; He swears that he 'll never go to sea any more The next time he wishes old England to see It will be when the railroad runs over the sea  Note: Alternate chorus I've heard is: Caterwaulin', Tarpaulin', Harpoonin' and all Tune is another Derry Down variant RG From Ballads and Sea Songs of Newfoundland, Greenleaf Collected from John Edison, Fleur de Lys, 1929 @sailor @fish Filename[ PADWHAL Play.exe DERRDWN2 RG ===DOCUMENT BOUNDARY===""",
      "vector_lyrics": [ ... ]
    }
  }

Remote Connector details:

_GET /_plugins/ml/connectors/WVgvuY8Ba1cmZFt-2qvW

{
  "name": "DevToolsRemoteConnector7",
  "version": "1",
  "description": "This is remote connector configurator created from Open Search Lib at runtime",
  "protocol": "http",
  "parameters": {
    "endpoint": "****************",
    "deployment_name": "*******",
    "temperature": "0.0",
    "model": "****",
    "api_version": "**********"
  },
  "actions": [
    {
      "action_type": "PREDICT",
      "method": "POST",
      "url": "https://${parameters.endpoint}/openai/deployments/${parameters.deployment_name}/chat/completions?api-version=${parameters.api_version}",
      "headers": {
        "api-key": "${credential.openAI_key}"
      },
      "request_body": """{ "messages": "${parameters.messages}", "temperature": ${parameters.temperature} }"""
    }
  ],
  "owner": {
    "name": "admin",
    "backend_roles": [
      "admin"
    ],
    "roles": [
      "own_index",
      "all_access"
    ],
    "custom_attribute_names": [],
    "user_requested_tenant": "__user__"
  },
  "access": "private"
}

Remote model details

_GET /_plugins/ml/models/XlgwuY8Ba1cmZFt-Nquz

{
    "name": "devtools_remote_model6",
    "model_group_id": "OCW-hY8BT-AsUikAscDk",
    "algorithm": "REMOTE",
    "model_version": "8",
    "description": "Description about the remote model",
    "model_state": "DEPLOYED",
    "created_time": 1716798895794,
    "last_updated_time": 1716798975296,
    "last_deployed_time": 1716798975296,
    "auto_redeploy_retry_times": 0,
    "planning_worker_node_count": 1,
    "current_worker_node_count": 1,
    "planning_worker_nodes": [
      "SgpCQj20SGWdpTIL0-bXGg"
    ],
    "deploy_to_all_nodes": true,
    "is_hidden": false,
    "connector_id": "WVgvuY8Ba1cmZFt-2qvW"
  }

Agent Configuration:

_GET /_plugins/ml/agents/ZVgwuY8Ba1cmZFt-3qvO

{
    "name": "Song Lyrics Agent opensearch-project/OpenSearch#1",
    "type": "flow",
    "description": "Description about the agent",
    "tools": [
      {
        "type": "VectorDBTool",
        "parameters": {
          "input": "${parameters.messages}",
          "source_field": """["artist","album","meta_data"]""",
          "embedding_field": "vector_lyrics",
          "index": "song_lyrics_index",
          "model_id": "cyEKpY8BQusrTe_CRBnv",
          "k": "4"
        },
        "include_output_in_agent_response": false
      },
      {
        "type": "MLModelTool",
        "description": "Tool for answering the question",
        "parameters": {
          "model_id": "XlgwuY8Ba1cmZFt-Nquz",
          "message": """This is the context:\n${parameters.VectorDBTool.output}\nThis is your question:\n${parameters.messages}\n. Give an answer based on the given context\n.""",
          "reponse_field": "result"
        },
        "include_output_in_agent_response": false
      }
    ],
    "created_time": 1716798938827,
    "last_updated_time": 1716798938827,
    "is_hidden": false
  }

Execute the agent with the below:

POST /_plugins/_ml/agents/ZVgwuY8Ba1cmZFt-3qvO/_execute
{
    "parameters" : {
        "messages" : "List the songs which are very peppy beat songs from  the lyrics"
    }
}

Following response is observed: "{\"error\":{\"reason\":\"Invalid Request\",\"details\":\"Error from remote service: \",\"type\":\"OpenSearchStatusException\"},\"status\":400}"

Expected behavior

As per the request made to agent, the syntensized response has to be the output. Instead error is observed.

Additional Details

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

uname -a

Darwin EPINCHEW00ED 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:49 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6020 arm64

Additional context Add any other context about the problem here.

peternied commented 1 month ago

[Triage - attendees 1 2 3 4 5 6 @admin-techheralds Thanks for creating this issue, it should be transfer to the ml-commons repo they can help you in that repo

@opensearch-project/admin could you please transfer this issue to ml-commons?

ylwu-amzn commented 1 month ago

@admin-techheralds I guess your model doesn't work. Can you test your model XlgwuY8Ba1cmZFt-Nquz and share predict API response ?

dblock commented 1 week ago

Catch All Triage - 1 2 3 4 5 6