azureopenai - embedding operation does not work with specified model

hussner commented 7 months ago

Describe the bug

Pretty much as the title says, I can't get to make it work while using Azure OpenAI.

Setup and configuration

*API Response Error::

File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 993, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'code': 'OperationNotSupported', 'message': 'The embeddings operation does not work with the specified model, gpt-35-turbo. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'}}

My config.toml and environment vars (be sure to redact API keys):

LLM_MODEL="gpt-3.5-turbo-1106"
LLM_API_KEY="xxx
LLM_EMBEDDING_MODEL="azureopenai"
LLM_BASE_URL="https://xxx.openai.azure.com"
LLM_DEPLOYMENT_NAME="chat-gpt-35-turbo"
LLM_API_VERSION="2023-05-15"
WORKSPACE_DIR="./workspace"

My model and agent (you can see these settings in the UI):

Model: gpt-35-turbo (azureopenai)
Agent:

Commands I ran to install and run OpenDevin:

make run

Steps to Reproduce: 1. 2. 3.

Logs, error messages, and screenshots: Running the app... Starting backend server... Waiting for the backend to start... /root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "modellist" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( /root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "modelname" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( /root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_groupalias" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( /root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "modelinfo" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( /root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "modelid" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = (). warnings.warn( INFO: Started server process [77916] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:3000 (Press CTRL+C to quit) Backend started successfully. Starting frontend with npm...

opendevin-frontend@0.1.0 start vite --port 3001 --host 0.0.0.0

VITE v5.2.8 ready in 437 ms

➜ Local: http://localhost:3001/ ➜ Network: http://172.16.1.4:3001/ ➜ press h + enter to show help INFO: ('127.0.0.1', 39008) - "WebSocket /ws?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzaWQiOiJlNWIzZGIzZC04YmI0LTQzMWItODI3Yy00NmIzMWRiNjc3ZGIifQ.3CwQZVrvb-AoJ8joU_7ZfzaG0lfog9acRnD40PuaAr8" [accepted] Starting loop_recv for sid: e5b3db3d-8bb4-431b-827c-46b31db677db, False INFO: connection open 21:45:44 - opendevin:INFO: sandbox.py:119 - Using workspace directory: /home/htadmin/OpenDevin/workspace 21:45:44 - opendevin:INFO: sandbox.py:332 - Container stopped 21:45:44 - opendevin:INFO: sandbox.py:367 - Container started 21:45:45 - opendevin:INFO: sandbox.py:383 - waiting for container to start: 1, container status: running 21:45:45 - opendevin:INFO: sandbox.py:210 - Connecting to opendevin@localhost via ssh. If you encounter any issues, you can try ssh -v -p 2222 opendevin@localhost with the password '5578fd09-9036-472d-af26-154b64c461fe' and report the issue on GitHub. 21:45:53 - opendevin:INFO: session.py:38 - WebSocket disconnected, sid: e5b3db3d-8bb4-431b-827c-46b31db677db INFO: connection closed INFO: ('127.0.0.1', 52864) - "WebSocket /ws?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzaWQiOiJlNWIzZGIzZC04YmI0LTQzMWItODI3Yy00NmIzMWRiNjc3ZGIifQ.3CwQZVrvb-AoJ8joU_7ZfzaG0lfog9acRnD40PuaAr8" [accepted] Starting loop_recv for sid: e5b3db3d-8bb4-431b-827c-46b31db677db, False INFO: connection open INFO: 127.0.0.1:52868 - "GET /refresh-files HTTP/1.1" 200 OK INFO: 127.0.0.1:52872 - "GET /messages/total HTTP/1.1" 200 OK INFO: 127.0.0.1:52870 - "GET /configurations HTTP/1.1" 200 OK 21:45:56 - opendevin:INFO: sandbox.py:119 - Using workspace directory: /home/htadmin/OpenDevin/workspace 21:45:57 - opendevin:INFO: sandbox.py:210 - Connecting to opendevin@localhost via ssh. If you encounter any issues, you can try ssh -v -p 2222 opendevin@localhost with the password 'c1ec05f0-b511-4644-8330-1b13f0013c0f' and report the issue on GitHub. 21:45:58 - opendevin:INFO: sandbox.py:119 - Using workspace directory: /home/htadmin/OpenDevin/workspace 21:45:58 - opendevin:INFO: sandbox.py:210 - Connecting to opendevin@localhost via ssh. If you encounter any issues, you can try ssh -v -p 2222 opendevin@localhost with the password '10553465-dea4-4864-a6ae-4f16b312bec2' and report the issue on GitHub. INFO: 127.0.0.1:52888 - "GET /refresh-files HTTP/1.1" 200 OK

============== STEP 0

PLAN: powershell script

ERROR: Error code: 400 - {'error': {'code': 'OperationNotSupported', 'message': 'The embeddings operation does not work with the specified model, gpt-35-turbo. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'}} Traceback (most recent call last): File "/home/htadmin/OpenDevin/opendevin/controller/agent_controller.py", line 135, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/htadmin/OpenDevin/agenthub/monologue_agent/agent.py", line 202, in step self._initialize(state.plan.main_goal) File "/home/htadmin/OpenDevin/agenthub/monologue_agent/agent.py", line 189, in _initialize self._add_event(action.to_dict()) File "/home/htadmin/OpenDevin/agenthub/monologue_agent/agent.py", line 121, in _add_event self.memory.add_event(event) File "/home/htadmin/OpenDevin/agenthub/monologue_agent/utils/memory.py", line 84, in add_event self.index.insert(doc) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/base.py", line 242, in insert self.insert_nodes(nodes, insert_kwargs) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 330, in insert_nodes self._insert(nodes, insert_kwargs) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 312, in _insert self._add_nodes_to_index(self._index_struct, nodes, insert_kwargs) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 233, in _add_nodes_to_index nodes_batch = self._get_node_with_embedding(nodes_batch, show_progress) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 141, in _get_node_with_embedding id_to_embed_map = embed_nodes( ^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/utils.py", line 138, in embed_nodes new_embeddings = embed_model.get_text_embedding_batch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 211, in wrapper result = func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py", line 326, in get_text_embedding_batch embeddings = self._get_text_embeddings(cur_batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/embeddings/openai/base.py", line 427, in _get_text_embeddings return get_embeddings( ^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 289, in wrapped_f return self(f, args, kw) ^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 379, in call do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 325, in iter raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 158, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 382, in call result = fn(*args, kwargs) ^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/embeddings/openai/base.py", line 180, in get_embeddings data = client.embeddings.create(input=list_of_text, model=engine, kwargs).data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/resources/embeddings.py", line 113, in create return self._post( ^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1213, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 902, in request return self._request( ^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 993, in _request raise self._make_status_error_from_response(err.response) from None openai.BadRequestError: Error code: 400 - {'error': {'code': 'OperationNotSupported', 'message': 'The embeddings operation does not work with the specified model, gpt-35-turbo. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'}} 21:46:11 - opendevin:ERROR: agent_controller.py:113 - Error in loop Traceback (most recent call last): File "/home/htadmin/OpenDevin/opendevin/controller/agent_controller.py", line 111, in start_loop finished = await self.step(i) ^^^^^^^^^^^^^^^^^^ File "/home/htadmin/OpenDevin/opendevin/controller/agent_controller.py", line 135, in step action = self.agent.step(self.state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/htadmin/OpenDevin/agenthub/monologue_agent/agent.py", line 202, in step self._initialize(state.plan.main_goal) File "/home/htadmin/OpenDevin/agenthub/monologue_agent/agent.py", line 189, in _initialize self._add_event(action.to_dict()) File "/home/htadmin/OpenDevin/agenthub/monologue_agent/agent.py", line 121, in _add_event self.memory.add_event(event) File "/home/htadmin/OpenDevin/agenthub/monologue_agent/utils/memory.py", line 84, in add_event self.index.insert(doc) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/base.py", line 242, in insert self.insert_nodes(nodes, insert_kwargs) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 330, in insert_nodes self._insert(nodes, insert_kwargs) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 312, in _insert self._add_nodes_to_index(self._index_struct, nodes, insert_kwargs) File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 233, in _add_nodes_to_index nodes_batch = self._get_node_with_embedding(nodes_batch, show_progress) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/vector_store/base.py", line 141, in _get_node_with_embedding id_to_embed_map = embed_nodes( ^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/indices/utils.py", line 138, in embed_nodes new_embeddings = embed_model.get_text_embedding_batch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 211, in wrapper result = func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py", line 326, in get_text_embedding_batch embeddings = self._get_text_embeddings(cur_batch) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/embeddings/openai/base.py", line 427, in _get_text_embeddings return get_embeddings( ^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 289, in wrapped_f return self(f, args, kw) ^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 379, in call do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 325, in iter raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 158, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/tenacity/init.py", line 382, in call result = fn(*args, kwargs) ^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/llama_index/embeddings/openai/base.py", line 180, in get_embeddings data = client.embeddings.create(input=list_of_text, model=engine, kwargs).data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/resources/embeddings.py", line 113, in create return self._post( ^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1213, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 902, in request return self._request( ^^^^^^^^^^^^^^ File "/root/.cache/pypoetry/virtualenvs/opendevin-dT9FkgQq-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 993, in _request raise self._make_status_error_from_response(err.response) from None openai.BadRequestError: Error code: 400 - {'error': {'code': 'OperationNotSupported', 'message': 'The embeddings operation does not work with the specified model, gpt-35-turbo. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'}}

Additional Context

hussner commented 7 months ago

Please guys? 😞

enyst commented 7 months ago

LLM_MODEL="gpt-3.5-turbo-1106" LLM_EMBEDDING_MODEL="azureopenai" LLM_BASE_URL="https://xxx.openai.azure.com" LLM_DEPLOYMENT_NAME="chat-gpt-35-turbo" LLM_API_VERSION="2023-05-15"

My model and agent (you can see these settings in the UI): Model: gpt-35-turbo (azureopenai)

This looks odd? I expected that the LLM_MODEL above should be the same with the Model below, and used in the call. In the call, it complains about "get-35-turbo", which perhaps doesn't exist or doesn't work?

Can you please make sure that the setting in the toml/env is indeed "got-3.5-turbo-1106", start the app, don't change any settings, start a task. Does it work?

Also, please run this and tell the result: git log -n1

hussner commented 7 months ago

LLM_MODEL="gpt-3.5-turbo-1106" LLM_EMBEDDING_MODEL="azureopenai" LLM_BASE_URL="https://xxx.openai.azure.com" LLM_DEPLOYMENT_NAME="chat-gpt-35-turbo" LLM_API_VERSION="2023-05-15"

My model and agent (you can see these settings in the UI): Model: gpt-35-turbo (azureopenai)

This looks odd? I expected that the LLM_MODEL above should be the same with the Model below, and used in the call. In the call, it complains about "get-35-turbo", which perhaps doesn't exist or doesn't work?

Can you please make sure that the setting in the toml/env is indeed "got-3.5-turbo-1106", start the app, don't change any settings, start a task. Does it work?

Also, please run this and tell the result: git log -n1

I understand but indeed my deployment name is that name so when the API identifies it, it has to match. Nonetheless, I have tried your theory, yet, I think I must have tried it before going through several combinations of toml config (none worked, obviously :D) and the message is still the same.

This Config: LLM_MODEL="chat-gpt-35-turbo" LLM_API_KEY="xxx" LLM_EMBEDDING_MODEL="azureopenai" LLM_BASE_URL="https://xxx.openai.azure.com/" LLM_DEPLOYMENT_NAME="chat-gpt-35-turbo" LLM_API_VERSION="2023-05-15" WORKSPACE_DIR="./workspace"

openai.BadRequestError: Error code: 400 - {'error': {'code': 'OperationNotSupported', 'message': 'The embeddings operation does not work with the specified model, gpt-35-turbo. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'}}

tobitege commented 7 months ago

See litellm docs: for Azure Open AI it uses different identifiers, like AZURE_API_KEY, not LLM_API_KEY? https://docs.litellm.ai/docs/providers/azure

hussner commented 7 months ago

See litellm docs: for Azure Open AI it uses different identifiers, like AZURE_API_KEY, not LLM_API_KEY? https://docs.litellm.ai/docs/providers/azure

After trying your solution it does not work. Though, that message is a message from API, so, the API responds but it does responds weird - no idea why it does say that. Can't find nothing related to this on the Internet.

openai.openaiError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

tobitege commented 7 months ago

See litellm docs: for Azure Open AI it uses different identifiers, like AZURE_API_KEY, not LLM_API_KEY? https://docs.litellm.ai/docs/providers/azure

After trying your solution it does not work. Though, that message is a message from API, so, the API responds but it does responds weird - no idea why it does say that. Can't find nothing related to this on the Internet.

openai.openaiError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

I think this is a missing feature for OpenDevin to cover Azure.

As a workaround you could add the corresponding ENV variables in e.g. your bash startup script or similar (I'm not a Linux guy)?

enyst commented 7 months ago

See litellm docs: for Azure Open AI it uses different identifiers, like AZURE_API_KEY, not LLM_API_KEY? https://docs.litellm.ai/docs/providers/azure

@tobitege we are using llama-index for embeddings actually, not LiteLLM, sorry for the confusion. The original error is from an embeddings call. https://github.com/OpenDevin/OpenDevin/blob/main/agenthub/monologue_agent/utils/memory.py#L27

We're missing something here. So @hussner about deployment, I actually remember something like that from Azure, but ... I'm not sure for embeddings, let's see.

the model for embeddings, we have "text-embedding-ada-002"
azure_deployment (on llama side) is set from deployment_name
azure_endpoint (on llama side) we set it from LLM_BASE_URL
api_version from LLM_API_VERSION

@hussner Am I reading this right, if you set something in "deployment" then the error complains about that ? It seems to ignore text-embedding-ada-002 ?

hussner commented 7 months ago

@hussner Am I reading this right, if you set something in "deployment" then the error complains about that ? It seems to ignore text-embedding-ada-002 ?

That's correct @enyst. I do have embeddings deployed in my Az Open AI Studio which is text-embedded-ada-002 but I do not mention it in any config/env string :). It does seem to ignore text-embedding-ada-002 fully.

Thing is that Deployment name for the embedding text-embedding-adda-002 is not the same with the gpt-35. Could this actually be?

Any clue about it?

Thanks for reply!

enyst commented 7 months ago

Then can we use that as llm deployment name here, if you're up for an attempt. Is it "text-embedded-ada-002", with "embedded" not "embedding"?

VishalHaibatpure1 commented 7 months ago

@enyst @hussner @tobitege Looks like few changes are required in OpenDevin to support Azure OpenAI. I was able to run OpenDevin with below changes+hacks

While creating litellm instance, API_Version param is not passed. I had to add few lines in llm.py to support this (Ref screenshot-1 below)
It looks like OpenDevin expecting two models: Chat completion (gpt-35-turbo) and embedding (text-embedding-ada-002 ). But config.toml allows only one deployment name. In Azure each model requires its own deployment (this does not apply in case of pure OpenAI as it only works based on Key and does not require deploymentname). If we just give "gpt-35-turbo" model and deployment name then it will start throwing "embedding operation does not work with specified model" and vise versa. For this I had to do below changes
- LLM_EMBEDDING_MODEL param is used to detect embedding strategy from different provider and it expects the value "azureopenai" in Azure case. Here I had to define new config "LLM_EMBEDDING_DEPLOYMENT_NAME" and update this value in config.toml with "text-embedding-ada-002" model deployment name. I had to update multiple files Config.py, memory.py and MakeFile to define and use this new config along with LLM_EMBEDDING_MODEL.

Screenshot-1 (Changes to fix API_Version issue)

ScreenShot-2 (config.toml with new deployment param for Embedding model)

ScreenShot-3 (addition of Embedding deployment config in config.py )

ScreenShot-4 (changes in MakeFile)

ScreenShot-5 (Referring Embedding deployment in memory.py)

enyst commented 7 months ago

@VishalHaibatpure1 Thanks for sharing, Azure can be complicated. Please feel free to submit a PR!

For both completion and embeddings I see that these libraries, LiteLLM and llama-index, read variables from the environment if they're not set specifically from us.

So I assume it can be done that way too. I am not sure which is best, but I'm pretty sure we need better documentation 😅

hussner commented 7 months ago

@VishalHaibatpure1 - wow :) it's working after doing the changes you proposed. You are great, sir!

Thanks!

hussner commented 7 months ago

@enyst @VishalHaibatpure1 not sure if you wanna add something else but feel free to close it if you're done.

Again, thanks for helping.

enyst commented 7 months ago

Thanks to your help, we are now adding documentation for Azure. https://github.com/OpenDevin/OpenDevin/pull/1035 Feel free to review it and let us know if it's insufficient.

All-Hands-AI / OpenHands