ollama doesn't work

[Yeqishen]

For other models, RouteLLM supports any provider that has an OpenAI-compatible interface, which includes a wide-range of both closed and open-source models running locally or in the cloud. Once you have an OpenAI-compatible endpoint, set the --alt-base-url and --alt-api-key flags to point to your endpoint. e.g. For Anyscale Endpoints,

I followed the instructions above to use my ollama, but it doesn't work at all. The error I get for requesting interface 6060/v1 still seems to point to OpenAI, am I doing something wrong?

Command: 'python -m routellm.openai_server --routers mf --config /Users/dali/PycharmProjects/simply_crawler/configuration/routellm_config.yaml --alt-base-url http://localhost:11434/v1/ --alt-api-key ollama'

The following is the error content:

Launching server with routers: ['mf'] INFO: Started server process [18063] INFO: Waiting for application startup. Loading mf: 100%|██████████| 1/1 [00:00<00:00, 1.54it/s] INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit) INFO: 127.0.0.1:58563 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 778, in app await route.handle(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle await self.app(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 79, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 74, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/openai_server.py", line 175, in create_chat_completion routed_model = route_fn(prompt, threshold, ROUTED_PAIR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 42, in route if self.calculate_strong_win_rate(prompt) >= threshold: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 235, in calculate_strong_win_rate winrate = self.model.pred_win_rate( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 124, in pred_win_rate logits = self.forward([model_a, model_b], prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 113, in forward OPENAI_CLIENT.embeddings.create(input=[prompt], model=self.embedding_model) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create return self._post( ^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1240, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request return self._request( ^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1020, in _request raise self._make_status_error_from_response(err.response) from None openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: ollama. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}} INFO: 127.0.0.1:58565 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 778, in app await route.handle(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle await self.app(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 79, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 74, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/openai_server.py", line 175, in create_chat_completion routed_model = route_fn(prompt, threshold, ROUTED_PAIR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 42, in route if self.calculate_strong_win_rate(prompt) >= threshold: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 235, in calculate_strong_win_rate winrate = self.model.pred_win_rate( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 124, in pred_win_rate logits = self.forward([model_a, model_b], prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 113, in forward OPENAI_CLIENT.embeddings.create(input=[prompt], model=self.embedding_model) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create return self._post( ^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1240, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request return self._request( ^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1020, in _request raise self._make_status_error_from_response(err.response) from None openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: ollama. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}} ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 778, in app await route.handle(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle await self.app(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 79, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 74, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/openai_server.py", line 175, in create_chat_completion routed_model = route_fn(prompt, threshold, ROUTED_PAIR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 42, in route if self.calculate_strong_win_rate(prompt) >= threshold: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 235, in calculate_strong_win_rate winrate = self.model.pred_win_rate( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 124, in pred_win_rate logits = self.forward([model_a, model_b], prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 113, in forward OPENAI_CLIENT.embeddings.create(input=[prompt], model=self.embedding_model) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create return self._post( ^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1240, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request return self._request( ^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1020, in _request raise self._make_status_error_from_response(err.response) from None openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: ollama. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}} INFO: 127.0.0.1:58566 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

[iojw]

Hi, it looks like what may be happening is that your request is being directed to the stronger model, which is GPT-4 by default, which is why it's trying to query the OpenAI endpoint.

You probably want to change your stronger model to another model that's available on ollama, which you can do using the --strong-model flag. For example,

python -m routellm.openai_server --routers mf --config /Users/dali/PycharmProjects/simply_crawler/configuration/routellm_config.yaml --alt-base-url http://localhost:11434/v1/ --alt-api-key ollama --strong-model llama3:70b --weak-model llama3

Let me know if this helps! I'll update the README to make this clearer as well.

[Yeqishen]

Thanks for the solution, I tried running the commands in your answer but got the same result.

Error: INFO: 127.0.0.1:64069 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__ await self.middleware_stack(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 778, in app await route.handle(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle await self.app(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 79, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/starlette/routing.py", line 74, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/openai_server.py", line 175, in create_chat_completion routed_model = route_fn(prompt, threshold, ROUTED_PAIR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 42, in route if self.calculate_strong_win_rate(prompt) >= threshold: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/routers.py", line 235, in calculate_strong_win_rate winrate = self.model.pred_win_rate( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 124, in pred_win_rate logits = self.forward([model_a, model_b], prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/routellm/routers/matrix_factorization/model.py", line 113, in forward OPENAI_CLIENT.embeddings.create(input=[prompt], model=self.embedding_model) File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/resources/embeddings.py", line 114, in create return self._post( ^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1240, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request return self._request( ^^^^^^^^^^^^^^ File "/Users/dali/PycharmProjects/simply_crawler/venv/lib/python3.11/site-packages/openai/_base_client.py", line 1020, in _request raise self._make_status_error_from_response(err.response) from None openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: ollama. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

[iojw]

@Yeqishen HI there, it looks like this is because you haven't set an OPENAI_API_KEY before launching the server. We require this for generating embeddings.

Let me know if this works for you.

[iojw]

@Yeqishen Closing for now, please let me know if you still face this issue!

[Yeqishen]

@Yeqishen HI there, it looks like this is because you haven't set an OPENAI_API_KEY before launching the server. We require this for generating embeddings.

Let me know if this works for you.

Thank you for your advice. I'm sure I set the OPENAI_API_KEY before launching the server. If it's not set, the operation command will throw an error: openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

CODE:

import os
from configuration.configuration import get_config_data
config_data = get_config_data()

base_url = config_data["base_url"]
api_key = config_data["api_key"]

os.environ["OPENAI_API_KEY"] = api_key
os.environ["OPENAI_API_BASE"] = base_url

command = f'python -m routellm.openai_server --routers mf --config /Users/dali/PycharmProjects/simply_crawler/configuration/routellm_config.yaml --alt-base-url {base_url} --alt-api-key {api_key}'
os.system(command)

When I tried using a third-party OPENAI API managed service, I encountered the same problem. No matter what I set, I ended up connecting to the official OPENAI connection. If possible, we hope you can add some basic examples to this project (including third-party OPENAI API services, ollama and other local LLM support).

Looking forward to your reply.

[Yeqishen]

@iojw Hi, I found the cause of the problem. If you need to use a third-party service, you must set OPENAI_API_KEY and OPENAI_BASE_URL. You may also add this to the description.

[iojw]

@Yeqishen Thank you for looking into this! The OPENAI_BASE_URL flag can be set using the --base-url argument - it is mentioned in the README but I agree it can be made clearer.