search

OpenInterpreter / open-interpreter

A natural language interface for computers
http://openinterpreter.com/
GNU Affero General Public License v3.0
52.63k stars 4.65k forks source link

Failed to work with custom model #902

Open xinghao-1210 opened 9 months ago

xinghao-1210 commented 9 months ago

Describe the bug

I was trying to load the mistral as custom model for open interpreter. I started by following the codes as described here https://docs.openinterpreter.com/language-model-setup/custom-models

But it keeps using gpt4 instead of the local model. Could you let me know how to use the local custom model?

Reproduce

  1. Setup the custom Mistral-7B-Instruct model and works well
  2. Followed https://docs.openinterpreter.com/language-model-setup/custom-models to use the same naive model
  3. interpreter.chat("hello world")

Expected behavior

It does not seem to use the naive model which should repeat what I input. gpt4 is used instead as shown in the following

Screenshots

Write messages to the user in Markdown. In general, try to make plans with as few steps as possible. As for actually executing code to carry out that plan, for stateful languages (like python, javascript, shell, but NOT for html which starts from 0 every time) it's critical not to try to do everything in one code block. You should try something, print information about it, then continue from there in tiny, informed steps. You will never get it on the first try, and attempting it in one go will often lead to errors you cant see. You are capable of any task.

    {'role': 'user', 'type': 'message', 'content': 'hello world'}

AuthenticationError Traceback (most recent call last) File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/llms/openai.py:344, in OpenAIChatCompletion.completion(self, model_response, timeout, model, messages, print_verbose, api_key, api_base, acompletion, logging_obj, optional_params, litellm_params, logger_fn, headers, custom_prompt_dict, client) 343 else: --> 344 raise e 345 except OpenAIError as e:

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/llms/openai.py:271, in OpenAIChatCompletion.completion(self, model_response, timeout, model, messages, print_verbose, api_key, api_base, acompletion, logging_obj, optional_params, litellm_params, logger_fn, headers, custom_prompt_dict, client) 270 elif optional_params.get("stream", False): --> 271 return self.streaming( 272 logging_obj=logging_obj, 273 headers=headers, 274 data=data, 275 model=model, 276 api_base=api_base, 277 api_key=api_key, 278 timeout=timeout, 279 client=client, 280 max_retries=max_retries, 281 ) 282 else:

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/llms/openai.py:440, in OpenAIChatCompletion.streaming(self, logging_obj, timeout, data, model, api_key, api_base, client, max_retries, headers) 430 logging_obj.pre_call( 431 input=data["messages"], 432 api_key=api_key, (...) 438 }, 439 ) --> 440 response = openai_client.chat.completions.create(**data, timeout=timeout) 441 streamwrapper = CustomStreamWrapper( 442 completion_stream=response, 443 model=model, 444 custom_llm_provider="openai", 445 logging_obj=logging_obj, 446 )

File ~/.conda/envs/llm/lib/python3.11/site-packages/openai/_utils/_utils.py:271, in required_args..inner..wrapper(*args, *kwargs) 270 raise TypeError(msg) --> 271 return func(args, **kwargs)

File ~/.conda/envs/llm/lib/python3.11/site-packages/openai/resources/chat/completions.py:643, in Completions.create(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout) 594 @required_args(["messages", "model"], ["messages", "model", "stream"]) 595 def create( 596 self, (...) 641 timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN, 642 ) -> ChatCompletion | Stream[ChatCompletionChunk]: --> 643 return self._post( 644 "/chat/completions", 645 body=maybe_transform( 646 { 647 "messages": messages, 648 "model": model, 649 "frequency_penalty": frequency_penalty, 650 "function_call": function_call, 651 "functions": functions, 652 "logit_bias": logit_bias, 653 "logprobs": logprobs, 654 "max_tokens": max_tokens, 655 "n": n, 656 "presence_penalty": presence_penalty, 657 "response_format": response_format, 658 "seed": seed, 659 "stop": stop, 660 "stream": stream, 661 "temperature": temperature, 662 "tool_choice": tool_choice, 663 "tools": tools, 664 "top_logprobs": top_logprobs, 665 "top_p": top_p, 666 "user": user, 667 }, 668 completion_create_params.CompletionCreateParams, 669 ), 670 options=make_request_options( 671 extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout 672 ), 673 cast_to=ChatCompletion, 674 stream=stream or False, 675 stream_cls=Stream[ChatCompletionChunk], 676 )

File ~/.conda/envs/llm/lib/python3.11/site-packages/openai/_base_client.py:1091, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls) 1088 opts = FinalRequestOptions.construct( 1089 method="post", url=path, json_data=body, files=to_httpx_files(files), **options 1090 ) -> 1091 return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File ~/.conda/envs/llm/lib/python3.11/site-packages/openai/_base_client.py:852, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls) 843 def request( 844 self, 845 cast_to: Type[ResponseT], (...) 850 stream_cls: type[_StreamT] | None = None, 851 ) -> ResponseT | _StreamT: --> 852 return self._request( 853 cast_to=cast_to, 854 options=options, 855 stream=stream, 856 stream_cls=stream_cls, 857 remaining_retries=remaining_retries, 858 )

File ~/.conda/envs/llm/lib/python3.11/site-packages/openai/_base_client.py:933, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls) 931 err.response.read() --> 933 raise self._make_status_error_from_response(err.response) from None 935 return self._process_response( 936 cast_to=cast_to, 937 options=options, (...) 940 stream_cls=stream_cls, 941 )

AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: x. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

During handling of the above exception, another exception occurred:

OpenAIError Traceback (most recent call last) File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/main.py:748, in completion(model, messages, timeout, temperature, top_p, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, deployment_id, functions, function_call, base_url, api_version, api_key, model_list, **kwargs) 742 logging.post_call( 743 input=messages, 744 api_key=api_key, 745 original_response=str(e), 746 additional_args={"headers": headers}, 747 ) --> 748 raise e 750 if optional_params.get("stream", False): 751 ## LOGGING

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/main.py:723, in completion(model, messages, timeout, temperature, top_p, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, deployment_id, functions, function_call, base_url, api_version, api_key, model_list, **kwargs) 722 try: --> 723 response = openai_chat_completions.completion( 724 model=model, 725 messages=messages, 726 headers=headers, 727 model_response=model_response, 728 print_verbose=print_verbose, 729 api_key=api_key, 730 api_base=api_base, 731 acompletion=acompletion, 732 logging_obj=logging, 733 optional_params=optional_params, 734 litellm_params=litellm_params, 735 logger_fn=logger_fn, 736 timeout=timeout, 737 custom_prompt_dict=custom_prompt_dict, 738 client=client, # pass AsyncOpenAI, OpenAI client 739 ) 740 except Exception as e: 741 ## LOGGING - log the original exception returned

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/llms/openai.py:350, in OpenAIChatCompletion.completion(self, model_response, timeout, model, messages, print_verbose, api_key, api_base, acompletion, logging_obj, optional_params, litellm_params, logger_fn, headers, custom_prompt_dict, client) 349 if hasattr(e, "status_code"): --> 350 raise OpenAIError(status_code=e.status_code, message=str(e)) 351 else:

OpenAIError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: x. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

During handling of the above exception, another exception occurred:

AuthenticationError Traceback (most recent call last) File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/llm/llm.py:221, in fixed_litellm_completions(params) 220 try: --> 221 yield from litellm.completion(params) 222 except: 223 # If the second attempt also fails, raise the first error

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:2129, in client..wrapper(*args, **kwargs) 2128 e.message += f"\n Check the log in your dashboard - {liteDebuggerClient.dashboard_url}" -> 2129 raise e

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:2036, in client..wrapper(*args, *kwargs) 2035 # MODEL CALL -> 2036 result = original_function(args, **kwargs) 2037 end_time = datetime.datetime.now()

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/main.py:1746, in completion(model, messages, timeout, temperature, top_p, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, deployment_id, functions, function_call, base_url, api_version, api_key, model_list, **kwargs) 1744 except Exception as e: 1745 ## Map to OpenAI Exception -> 1746 raise exception_type( 1747 model=model, 1748 custom_llm_provider=custom_llm_provider, 1749 original_exception=e, 1750 completion_kwargs=args, 1751 )

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:6605, in exception_type(model, original_exception, custom_llm_provider, completion_kwargs) 6604 if exception_mapping_worked: -> 6605 raise e 6606 else:

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:5566, in exception_type(model, original_exception, custom_llm_provider, completion_kwargs) 5565 exception_mapping_worked = True -> 5566 raise AuthenticationError( 5567 message=f"OpenAIException - {original_exception.message}", 5568 llm_provider="openai", 5569 model=model, 5570 response=original_exception.response, 5571 ) 5572 elif original_exception.status_code == 404:

AuthenticationError: OpenAIException - Error code: 401 - {'error': {'message': 'Incorrect API key provided: x. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

During handling of the above exception, another exception occurred:

APIError Traceback (most recent call last) Cell In[11], line 1 ----> 1 interpreter.chat("Hi! Can you print hello world?")

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/core.py:86, in OpenInterpreter.chat(self, message, display, stream) 83 initial_messagecount = len(self.messages) 85 # If stream=False, pull from the stream. ---> 86 for in self._streaming_chat(message=message, display=display): 87 pass 89 # Return new messages

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/core.py:113, in OpenInterpreter._streaming_chat(self, message, display) 107 def _streaming_chat(self, message=None, display=True): 108 # Sometimes a little more code -> a much better experience! 109 # Display mode actually runs interpreter.chat(display=False, stream=True) from within the terminal_interface. 110 # wraps the vanilla .chat(display=False) generator in a display. 111 # Quite different from the plain generator stuff. So redirect to that 112 if display: --> 113 yield from terminal_interface(self, message) 114 return 116 # One-off message

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/terminal_interface/terminal_interface.py:135, in terminal_interface(interpreter, message) 127 message = { 128 "role": "user", 129 "type": "image", 130 "format": "path", 131 "content": image_path, 132 } 134 try: --> 135 for chunk in interpreter.chat(message, display=False, stream=True): 136 yield chunk 138 # Is this for thine eyes?

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/core.py:148, in OpenInterpreter._streaming_chat(self, message, display) 134 self.messages = message 136 # DISABLED because I think we should just not transmit images to non-multimodal models? 137 # REENABLE this when multimodal becomes more common: 138 (...) 146 147 # This is where it all happens! --> 148 yield from self._respond_and_store() 150 # Save conversation if we've turned conversation_history on 151 if self.conversation_history: 152 # If it's the first message, set the conversation name

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/core.py:194, in OpenInterpreter._respond_and_store(self) 190 return "format" in chunk and chunk["format"] == "active_line" 192 last_flag_base = None --> 194 for chunk in respond(self): 195 if chunk["content"] == "": 196 continue

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/respond.py:49, in respond(interpreter) 46 ### RUN THE LLM ### 48 try: ---> 49 for chunk in interpreter.llm.run(messages_for_llm): 50 yield {"role": "assistant", **chunk} 52 except litellm.exceptions.BudgetExceededError:

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/llm/llm.py:193, in Llm.run(self, messages) 191 yield from run_function_calling_llm(self, params) 192 else: --> 193 yield from run_text_llm(self, params)

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/llm/run_text_llm.py:19, in run_text_llm(llm, params) 16 accumulated_block = "" 17 language = None ---> 19 for chunk in llm.completions(**params): 20 if llm.interpreter.verbose: 21 print("Chunk in coding_llm", chunk)

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/llm/llm.py:224, in fixed_litellm_completions(params) 221 yield from litellm.completion(params) 222 except: 223 # If the second attempt also fails, raise the first error --> 224 raise first_error

File ~/.conda/envs/llm/lib/python3.11/site-packages/interpreter/core/llm/llm.py:205, in fixed_litellm_completions(params) 203 first_error = None 204 try: --> 205 yield from litellm.completion(params) 206 except Exception as e: 207 # Store the first error 208 first_error = e

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:2129, in client..wrapper(*args, **kwargs) 2125 if ( 2126 liteDebuggerClient and liteDebuggerClient.dashboard_url != None 2127 ): # make it easy to get to the debugger logs if you've initialized it 2128 e.message += f"\n Check the log in your dashboard - {liteDebuggerClient.dashboard_url}" -> 2129 raise e

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:2036, in client..wrapper(*args, *kwargs) 2034 return cached_result 2035 # MODEL CALL -> 2036 result = original_function(args, **kwargs) 2037 end_time = datetime.datetime.now() 2038 if "stream" in kwargs and kwargs["stream"] == True: 2039 # TODO: Add to cache for streaming

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/main.py:1746, in completion(model, messages, timeout, temperature, top_p, n, stream, stop, max_tokens, presence_penalty, frequency_penalty, logit_bias, user, response_format, seed, tools, tool_choice, logprobs, top_logprobs, deployment_id, functions, function_call, base_url, api_version, api_key, model_list, **kwargs) 1743 return response 1744 except Exception as e: 1745 ## Map to OpenAI Exception -> 1746 raise exception_type( 1747 model=model, 1748 custom_llm_provider=custom_llm_provider, 1749 original_exception=e, 1750 completion_kwargs=args, 1751 )

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:6605, in exception_type(model, original_exception, custom_llm_provider, completion_kwargs) 6603 # don't let an error with mapping interrupt the user from receiving an error from the llm api calls 6604 if exception_mapping_worked: -> 6605 raise e 6606 else: 6607 raise original_exception

File ~/.conda/envs/llm/lib/python3.11/site-packages/litellm/utils.py:5620, in exception_type(model, original_exception, custom_llm_provider, completion_kwargs) 5618 else: 5619 exception_mapping_worked = True -> 5620 raise APIError( 5621 status_code=original_exception.status_code, 5622 message=f"OpenAIException - {original_exception.message}", 5623 llm_provider="openai", 5624 model=model, 5625 request=original_exception.request, 5626 ) 5627 else: 5628 # if no status code then it is an APIConnectionError: https://github.com/openai/openai-python#handling-errors 5629 raise APIConnectionError( 5630 cause=original_exception.cause, 5631 llm_provider=custom_llm_provider, 5632 model=model, 5633 request=original_exception.request, 5634 )

APIError: OpenAIException - The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

Open Interpreter version

0.20

Python version

3.11

Operating System name and version

amazon linux 2

Additional context

No response

donymorph commented 9 months ago

image

Notnaton commented 9 months ago

@xinghao-1210 Can you share the config/command you used to run interpreter?

xinghao-1210 commented 9 months ago

@xinghao-1210 Can you share the config/command you used to run interpreter?

I am running it through Amazon EC2 instance linux system. python 3.11, interpreter v0.2.0

Follow https://docs.openinterpreter.com/language-model-setup/custom-models, I first tried implementing my own toy model which doesn't work. And then I simply follow the codes from the link:

def custom_language_model(openai_message): """ OpenAI-compatible completions function (this one just echoes what the user said back). """ users_content = openai_message[-1].get("content") # Get last message's content

# To make it OpenAI-compatible, we yield this first:
yield {"delta": {"role": "assistant"}}

for character in users_content:
    yield {"delta": {"content": character}}

interpreter.llm.completion = custom_language_model

interpreter.llm.context_window = 2000 # In tokens interpreter.llm.max_tokens = 1000 # In tokens interpreter.llm.supports_vision = False # Does this completions endpoint accept images? interpreter.llm.supports_functions = False # Does this completions endpoint accept/return function calls?

interpreter.chat("Hi!") # Returns/displays "Hi!" character by character

I expect to get the interpreter repeat what is typed in according to the instruction. But it keeps using gpt-4. I have tried set api_key with my openai api but I don't mean to use gpt4/chatgpt. Also played around the interpreter.llm.offline/model/api but no success yet.

New to the llm world and appreciate the help

Notnaton commented 9 months ago

can you try setting interpreter.model = "openai/custom"

xinghao-1210 commented 9 months ago

can you try setting interpreter.model = "openai/custom"

edited the model as "openai/custom" It's still using Model: gpt-4 APIError: OpenAIException - The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

Added the dummy api_key: interpreter.llm.api_key = "fake_key" AuthenticationError: There might be an issue with your API key(s).

To reset your API key (we'll use OPENAI_API_KEY for this example, but you may need to reset your ANTHROPIC_API_KEY, HUGGINGFACE_API_KEY, etc): Mac/Linux: 'export OPENAI_API_KEY=your-key-here', Windows: 'setx OPENAI_API_KEY your-key-here' then restart terminal.

redthing1 commented 7 months ago

I have the same issue.

MikeBirdTech commented 7 months ago

Hi @redthing1 What is the code that you're running to have this issue?