CharlieJCJ commented 1 week ago

Related Issues: #74

Changes:

Included two example scripts, illustrating litellm usage in prompting and structured output modes. /examples/litellm_recipe_prompting.py /examples/litellm_recipe_structured_output.py (Note: need to put OpenAI and Anthropic keys in environment).
Included a backend parameter when using prompter code link, right now defaults to OpenAI.
Integrated with instructor for structure output support for a large coverage of models. For models that can't use litellm + instructor (structured output), used a try except block before dataset generation to check whether instructor works on a simple example code link.
Added time and cost logging (cost using litellm.completion_cost, if model cost is in the community-maintained mapping here)
Added estimate_total_tokens that includes estimate_output_tokens which derives from get_max_tokens that gets max output token of the specified model. code link
Using the same async, retry strategy as OpenAI Online Request Processor
Get litellm rate limit through hidden params dict, including x-ratelimit-limit-requests, and x-ratelimit-limit-tokens for rpm and tpm.

Future Works:

Refactoring of OpenAI Online request processor and Litellm Online such that both share the same functions will place in a base online request handler.
Support a baseline structured output strategy for models from inference platforms that do not support structured outputs from litellm. Right now, will directly prompt user that the current model doesn't support structured output. Note that litellm structured output model / provider coverage isn't good, link. Need additional research.
More performance optimization for auto rate limiting / retry strategies, need battle testing & experiments and comparisons.

Example curator-viewer's view:

Tested on the following models, all working for litellm + instructor structured output.

"claude-3-5-sonnet-20240620", # https://docs.litellm.ai/docs/providers/anthropic # anthropic has a different hidden param tokens structure. 
"claude-3-5-haiku-20241022",
"claude-3-haiku-20240307",
"claude-3-opus-20240229",
"claude-3-sonnet-20240229",
"gpt-4o-mini", # https://docs.litellm.ai/docs/providers/openai
"gpt-4o-2024-08-06",
"gpt-4-0125-preview",
"gpt-3.5-turbo-1106",
"gemini/gemini-1.5-flash", # https://docs.litellm.ai/docs/providers/gemini; https://ai.google.dev/gemini-api/docs/models # 20-30 iter/s
"gemini/gemini-1.5-pro", # 20-30 iter/s
"together_ai/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", # https://docs.together.ai/docs/serverless-models
"together_ai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
"together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1",

Note that the following models does not support structured output (i.e. response_format in Prompter)

# "together_ai/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF", # instructor not supported
# "deepinfra/nvidia/Llama-3.1-Nemotron-70B-Instruct" # instructor not supported

CharlieJCJ commented 1 week ago

Works for Claude model_name="claude-3-opus-20240229"

CharlieJCJ commented 1 week ago

After #149's merge (that resolves #145), do performance comparison of litellm vs. OpenAI request processors

CharlieJCJ commented 1 week ago

159 is been merged, now costs have been appropriately logged. litellm also supports cost logging now.

CharlieJCJ commented 1 week ago

TODO

Need to investigate why same prompt yield different prompt token issue #166

CharlieJCJ commented 1 week ago

Need to add a better default timeout.

CharlieJCJ commented 1 week ago

asyncio issue pops up

(bespokelabs-curator-py3.12) (base) ➜  bella git:(CURATOR-28-add-a-lite-llm-backend-for-curator) ✗ python examples/litellm_example.py 
Processing LiteLLM requests:  98%|█████████████████████████████████████████████████████████████████████████████████████▎ | 98/100 [00:12<00:00,  8.06it/s]
                                        title  ... cuisine_type
0                   \"Simple Pork and Rice\"   ...      Chinese
1   \"Simple Chicken and Vegetable Stir-fry\"  ...      Chinese
2             \"Ginger Chicken and Broccoli\"  ...      Chinese
3                        Simple Pork and Rice  ...      Chinese
4             \"Ginger Chicken and Broccoli\"  ...      Chinese
..                                        ...  ...          ...
95            \"Ginger Chicken and Broccoli\"  ...      Chinese
96              \"Simple Chicken Fried Rice\"  ...      Chinese
97            Chinese Pork and Egg Fried Rice  ...      Chinese
98     Garlic Ginger Bok Choy and Carrot Rice  ...      Chinese
99          \"Simple Pork and Rice Stir-fry\"  ...      Chinese

[100 rows x 4 columns]
Processing LiteLLM requests:   0%|                                                                                                | 0/100 [00:00<?, ?it/s]
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

2024-11-22 01:17:48,949 - bespokelabs.curator.request_processor.litellm_online_request_processor - ERROR - Error in request 0: litellm.APIConnectionError: Event loop is closed
Traceback (most recent call last):
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/main.py", line 481, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1251, in async_completion
    response = await client.post(api_base, headers=headers, json=request_body)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 166, in post
    raise e
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 123, in post
    response = await self.client.send(req, stream=stream)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1674, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1776, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 135, in handle_async_request
    await self._response_closed()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 250, in _response_closed
    await self.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 258, in aclose
    await self._network_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 53, in aclose
    await self._stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/streams/tls.py", line 201, in aclose
    await self.transport_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1287, in aclose
    self._transport.close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 1210, in close
    super().close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 875, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

2024-11-22 01:17:49,074 - bespokelabs.curator.request_processor.litellm_online_request_processor - ERROR - Error in request 1: litellm.APIConnectionError: Event loop is closed
Traceback (most recent call last):
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/main.py", line 481, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1251, in async_completion
    response = await client.post(api_base, headers=headers, json=request_body)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 166, in post
    raise e
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 123, in post
    response = await self.client.send(req, stream=stream)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1674, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1776, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 135, in handle_async_request
    await self._response_closed()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 250, in _response_closed
    await self.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 258, in aclose
    await self._network_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 53, in aclose
    await self._stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/streams/tls.py", line 201, in aclose
    await self.transport_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1287, in aclose
    self._transport.close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 1210, in close
    super().close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 875, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

2024-11-22 01:17:49,199 - bespokelabs.curator.request_processor.litellm_online_request_processor - ERROR - Error in request 0: litellm.APIConnectionError: Event loop is closed
Traceback (most recent call last):
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/main.py", line 481, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1251, in async_completion
    response = await client.post(api_base, headers=headers, json=request_body)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 166, in post
    raise e
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 123, in post
    response = await self.client.send(req, stream=stream)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1674, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1776, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 135, in handle_async_request
    await self._response_closed()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 250, in _response_closed
    await self.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 258, in aclose
    await self._network_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 53, in aclose
    await self._stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/streams/tls.py", line 201, in aclose
    await self.transport_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1287, in aclose
    self._transport.close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 1210, in close
    super().close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 875, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

2024-11-22 01:17:49,321 - bespokelabs.curator.request_processor.litellm_online_request_processor - ERROR - Error in request 1: litellm.APIConnectionError: Event loop is closed
Traceback (most recent call last):
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/main.py", line 481, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1251, in async_completion
    response = await client.post(api_base, headers=headers, json=request_body)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 166, in post
    raise e
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 123, in post
    response = await self.client.send(req, stream=stream)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1674, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1776, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 135, in handle_async_request
    await self._response_closed()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 250, in _response_closed
    await self.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 258, in aclose
    await self._network_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 53, in aclose
    await self._stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/streams/tls.py", line 201, in aclose
    await self.transport_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1287, in aclose
    self._transport.close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 1210, in close
    super().close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 875, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

2024-11-22 01:17:49,518 - bespokelabs.curator.request_processor.litellm_online_request_processor - ERROR - Error in request 0: litellm.APIConnectionError: Event loop is closed
Traceback (most recent call last):
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/main.py", line 481, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/vertex_ai_and_google_ai_studio/gemini/vertex_and_google_ai_studio_gemini.py", line 1251, in async_completion
    response = await client.post(api_base, headers=headers, json=request_body)  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 166, in post
    raise e
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/litellm/llms/custom_httpx/http_handler.py", line 123, in post
    response = await self.client.send(req, stream=stream)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1674, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1702, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1739, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1776, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py", line 377, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/connection.py", line 103, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 135, in handle_async_request
    await self._response_closed()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 250, in _response_closed
    await self.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py", line 258, in aclose
    await self._network_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 53, in aclose
    await self._stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/streams/tls.py", line 201, in aclose
    await self.transport_stream.aclose()
  File "/home/charlieji/.cache/pypoetry/virtualenvs/bespokelabs-curator-809-Lr9v-py3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1287, in aclose
    self._transport.close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 1210, in close
    super().close()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/selector_events.py", line 875, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 795, in call_soon
    self._check_closed()
  File "/home/charlieji/miniconda3/lib/python3.12/asyncio/base_events.py", line 541, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Processing LiteLLM requests:  99%|██████████████████████████████████████████████████████████████████████████████████████▏| 99/100 [00:16<00:00,  5.91it/s]
                                title                                        ingredients                                       instructions cuisine_type
0              Pan-Fried Pork Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine pork, scallions, soy...      Chinese
1                 Crispy Pork Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine ground pork, scallio...      Chinese
2                 Crispy Pork Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine ground pork, scallio...      Chinese
3          Crispy Golden Pork Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine pork, scallions, soy...      Chinese
4   Stir-Fried Pork with Green Pepper  [1 tbsp vegetable oil, 1 lb ground pork, 1/2 c...  [Heat vegetable oil in a wok or large skillet ...      Chinese
..                                ...                                                ...                                                ...          ...
95               Crispy Fried Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine ground pork, scallio...      Chinese
96       Simple Pork Wontons in Broth  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine ground pork, scallio...      Chinese
97               Crispy Fried Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine ground pork, scallio...      Chinese
98          Crispy Fried Pork Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine ground pork, scallio...      Chinese
99                Golden Pork Wontons  [1 lb ground pork, 1/2 cup chopped scallions, ...  [In a large bowl, combine the ground pork, sca...      Chinese

[100 rows x 4 columns]
(bespokelabs-curator-py3.12) (base) ➜  bella git:(CURATOR-28-add-a-lite-llm-backend-for-curator) ✗

CharlieJCJ commented 1 week ago

Need to fallback to prompt engineering -> json if the model doesn't have function calling / structured output

CharlieJCJ commented 1 day ago

Tested on the following models, all working for litellm + instructor structured output

"claude-3-5-sonnet-20240620", # https://docs.litellm.ai/docs/providers/anthropic # anthropic has a different hidden param tokens structure. 
"claude-3-5-haiku-20241022",
"claude-3-haiku-20240307",
"claude-3-opus-20240229",
"claude-3-sonnet-20240229",
"gpt-4o-mini", # https://docs.litellm.ai/docs/providers/openai
"gpt-4o-2024-08-06",
"gpt-4-0125-preview",
"gpt-3.5-turbo-1106",
"gemini/gemini-1.5-flash", # https://docs.litellm.ai/docs/providers/gemini; https://ai.google.dev/gemini-api/docs/models # 20-30 iter/s
"gemini/gemini-1.5-pro", # 20-30 iter/s
"together_ai/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", # https://docs.together.ai/docs/serverless-models
"together_ai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
"together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1",

bespokelabsai / curator

Add LiteLLM+instructor (for structured output) backend for curator #141

159 is been merged, now costs have been appropriately logged. litellm also supports cost logging now.