search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
14.37k stars 1.69k forks source link

[Bug]: return correct finish_reason for google ai studio content_filter on streaming #6104

Closed bachya closed 2 weeks ago

bachya commented 1 month ago

What happened?

Using litellm 1.48.18. This simple code:

import asyncio
import os

from litellm import acompletion

os.environ["GEMINI_API_KEY"] = "REDACTED"
os.environ["LITELLM_LOG"] = "DEBUG"

async def test() -> None:
    response = await acompletion(
        model="gemini/gemini-1.5-pro",
        messages=[
            {
                "role": "user",
                "content": "What are the downsides of prompt-hacking? For example, if you have a customer portal that is helping answer questions about the companies products, what are the possible risks here?",
            }
        ],
        stream=True,
    )
    async for chunk in response:
        print(chunk)

asyncio.run(test())

...will regularly return inconsistent results: sometimes (although irregularly), I'll get a complete response, but other times (more consistently), the stream will stop prematurely. See below for outputs.

Relevant log output

Example of truncated response:

Request to litellm:
litellm.acompletion(model='gemini/gemini-1.5-pro', messages=[{'role': 'user', 'content': 'What are the downsides fo prompt-hacking? For example, if you have a customer portal that is helping answer questions about the companies products, what are the possible risks here?'}], stream=True)

ASYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache'): None
Final returned optional params: {'stream': True}

POST Request Sent from LiteLLM:
curl -X POST \
https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:streamGenerateContent?key=AIzaSyCNKRChUQbannoALliThsbK2LORCY-kYmY&alt=sse \
-H 'Content-Type: *****' \
-d '{'gemini_api_key': 'AIzaSyCNKRChUQbannoALliThsbK2LORCY-kYmY', 'messages': [{'role': 'user', 'content': 'What are the downsides fo prompt-hacking? For example, if you have a customer portal that is helping answer questions about the companies products, what are the possible risks here?'}], 'api_base': None, 'model': 'gemini-1.5-pro', 'client': None, 'timeout': 600.0, 'extra_headers': None, 'optional_params': {}, 'logging_obj': <litellm.litellm_core_utils.litellm_logging.Logging object at 0x108108e30>, 'custom_llm_provider': 'gemini', 'litellm_params': {'acompletion': True, 'api_key': None, 'force_timeout': 600, 'logger_fn': None, 'verbose': False, 'custom_llm_provider': 'gemini', 'api_base': 'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:streamGenerateContent?key=*****=sse', 'litellm_call_id': 'a46e7af6-7440-406b-9eca-8f9fdee4ee84', 'model_alias_map': {}, 'completion_call_id': None, 'metadata': None, 'model_info': None, 'proxy_server_request': None, 'preset_cache_key': None, 'no-log': False, 'stream_response': {}, 'input_cost_per_token': None, 'input_cost_per_second': None, 'output_cost_per_token': None, 'output_cost_per_second': None, 'cooldown_time': None, 'text_completion': None, 'azure_ad_token_provider': None, 'user_continue_message': None, 'base_model': None}}'

RAW RESPONSE:
first stream response received

value of async chunk: {'text': '##', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 1, 'total_tokens': 38}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '##', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 1, 'total_tokens': 38}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': '##', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 1, 'total_tokens': 38}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': '##'}
self.sent_first_chunk: False
hold - False, model_response_str - ##
returning model_response: ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=1, prompt_tokens=37, total_tokens=38, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=1, prompt_tokens=37, total_tokens=38, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly'}
self.sent_first_chunk: True
hold - False, model_response_str -  Downsides of Prompt Hacking in a Customer Portal

While prompt engineering can be incredibly
returning model_response: ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=17, prompt_tokens=37, total_tokens=54, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=17, prompt_tokens=37, total_tokens=54, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt Hacking in a Customer Portal\n\nWhile prompt engineering can be incredibly', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': '', 'tool_use': None, 'is_finished': False, 'finish_reason': 'content_filter', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'tool_use': None, 'is_finished': False, 'finish_reason': 'content_filter', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': '', 'tool_use': None, 'is_finished': False, 'finish_reason': 'content_filter', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Success Call: Cache_hit=False
ModelResponse(id='chatcmpl-515e353e-45c1-4019-a4cf-eb2ef6a7774a', choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(content=None, role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337383, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
Goes into checking if chunk has hiddden created at param
Chunks have a created at hidden param
Chunks sorted
Looking up model=gemini/gemini-1.5-pro in model_cost_map
Logging Details LiteLLM-Async Success Call, cache_hit=False
Goes into checking if chunk has hiddden created at param
Chunks have a created at hidden param
Chunks sorted
Async success callbacks: Got a complete streaming response
Looking up model=gemini/gemini-1.5-pro in model_cost_map

Example of complete response:

Request to litellm:
litellm.acompletion(model='gemini/gemini-1.5-pro', messages=[{'role': 'user', 'content': 'What are the downsides fo prompt-hacking? For example, if you have a customer portal that is helping answer questions about the companies products, what are the possible risks here?'}], stream=True)

ASYNC kwargs[caching]: False; litellm.cache: None; kwargs.get('cache'): None
Final returned optional params: {'stream': True}

POST Request Sent from LiteLLM:
curl -X POST \
https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:streamGenerateContent?key=AIzaSyCNKRChUQbannoALliThsbK2LORCY-kYmY&alt=sse \
-H 'Content-Type: *****' \
-d '{'gemini_api_key': 'AIzaSyCNKRChUQbannoALliThsbK2LORCY-kYmY', 'messages': [{'role': 'user', 'content': 'What are the downsides fo prompt-hacking? For example, if you have a customer portal that is helping answer questions about the companies products, what are the possible risks here?'}], 'api_base': None, 'model': 'gemini-1.5-pro', 'client': None, 'timeout': 600.0, 'extra_headers': None, 'optional_params': {}, 'logging_obj': <litellm.litellm_core_utils.litellm_logging.Logging object at 0x106bf8770>, 'custom_llm_provider': 'gemini', 'litellm_params': {'acompletion': True, 'api_key': None, 'force_timeout': 600, 'logger_fn': None, 'verbose': False, 'custom_llm_provider': 'gemini', 'api_base': 'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:streamGenerateContent?key=*****=sse', 'litellm_call_id': '477c0f35-395b-437e-b75a-e87767f6791c', 'model_alias_map': {}, 'completion_call_id': None, 'metadata': None, 'model_info': None, 'proxy_server_request': None, 'preset_cache_key': None, 'no-log': False, 'stream_response': {}, 'input_cost_per_token': None, 'input_cost_per_second': None, 'output_cost_per_token': None, 'output_cost_per_second': None, 'cooldown_time': None, 'text_completion': None, 'azure_ad_token_provider': None, 'user_continue_message': None, 'base_model': None}}'

RAW RESPONSE:
first stream response received

value of async chunk: {'text': '##', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 1, 'total_tokens': 38}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '##', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 1, 'total_tokens': 38}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': '##', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 1, 'total_tokens': 38}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': '##'}
self.sent_first_chunk: False
hold - False, model_response_str - ##
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=1, prompt_tokens=37, total_tokens=38, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=1, prompt_tokens=37, total_tokens=38, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='##', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering'}
self.sent_first_chunk: True
hold - False, model_response_str -  Downsides of Prompt-Hacking in a Customer Portal

While prompt engineering
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=17, prompt_tokens=37, total_tokens=54, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=17, prompt_tokens=37, total_tokens=54, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' Downsides of Prompt-Hacking in a Customer Portal \n\nWhile prompt engineering', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337440, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' can be a powerful tool for improving chatbot responses in a customer portal, prompt-', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 33, 'total_tokens': 70}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' can be a powerful tool for improving chatbot responses in a customer portal, prompt-', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 33, 'total_tokens': 70}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' can be a powerful tool for improving chatbot responses in a customer portal, prompt-', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 33, 'total_tokens': 70}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' can be a powerful tool for improving chatbot responses in a customer portal, prompt-'}
self.sent_first_chunk: True
hold - False, model_response_str -  can be a powerful tool for improving chatbot responses in a customer portal, prompt-
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can be a powerful tool for improving chatbot responses in a customer portal, prompt-', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=33, prompt_tokens=37, total_tokens=70, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can be a powerful tool for improving chatbot responses in a customer portal, prompt-', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=33, prompt_tokens=37, total_tokens=70, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can be a powerful tool for improving chatbot responses in a customer portal, prompt-', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can be a powerful tool for improving chatbot responses in a customer portal, prompt-', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': 'hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 65, 'total_tokens': 102}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': 'hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 65, 'total_tokens': 102}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': 'hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 65, 'total_tokens': 102}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': 'hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **'}
self.sent_first_chunk: True
hold - False, model_response_str - hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:

**For Your Business:**

* **
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=65, prompt_tokens=37, total_tokens=102, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=65, prompt_tokens=37, total_tokens=102, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='hacking, which exploits loopholes in the system without considering ethical implications or long-term consequences, poses several risks:\n\n**For Your Business:**\n\n* **', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337441, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': 'Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 97, 'total_tokens': 134}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': 'Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 97, 'total_tokens': 134}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': 'Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 97, 'total_tokens': 134}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': 'Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,'}
self.sent_first_chunk: True
hold - False, model_response_str - Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337442, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=97, prompt_tokens=37, total_tokens=134, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337442, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=97, prompt_tokens=37, total_tokens=134, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337442, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='Inaccurate & Irrelevant Information:** Hacked prompts can trick the system into providing inaccurate, irrelevant, or even fabricated information. This damages customer trust, creates frustration,', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337442, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 145, 'total_tokens': 182}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 145, 'total_tokens': 182}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 145, 'total_tokens': 182}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your'}
self.sent_first_chunk: True
hold - False, model_response_str -  and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.
* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337443, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=145, prompt_tokens=37, total_tokens=182, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337443, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=145, prompt_tokens=37, total_tokens=182, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337443, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' and can lead to negative brand perception. Imagine a user asking about your refund policy and getting an incorrect answer because someone manipulated the system.\n* **Data Leakage and Security Risks:** Clever prompt-hacking could potentially extract sensitive data from your', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337443, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 193, 'total_tokens': 230}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 193, 'total_tokens': 230}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 193, 'total_tokens': 230}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose'}
self.sent_first_chunk: True
hold - False, model_response_str -  knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.
* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337444, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=193, prompt_tokens=37, total_tokens=230, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337444, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=193, prompt_tokens=37, total_tokens=230, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337444, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' knowledge base or bypass security measures. This could expose confidential company information or even customer data, leading to legal and reputational damage.\n* **Erosion of Trust in the System:** If users realize the system is easily manipulated, they will lose', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337444, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': " trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 257, 'total_tokens': 294}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': " trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 257, 'total_tokens': 294}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': " trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 257, 'total_tokens': 294}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': " trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user"}
self.sent_first_chunk: True
hold - False, model_response_str -  trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.
* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337445, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=257, prompt_tokens=37, total_tokens=294, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337445, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=257, prompt_tokens=37, total_tokens=294, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337445, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" trust in its ability to provide reliable information. This can lead to decreased usage of the customer portal and reliance on less efficient support channels.\n* **Increased Maintenance & Development Costs:** Constantly having to identify and patch vulnerabilities created by prompt-hacking will divert resources away from improving the chatbot's core functionality and user", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337445, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 321, 'total_tokens': 358}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 321, 'total_tokens': 358}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 321, 'total_tokens': 358}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation'}
self.sent_first_chunk: True
hold - False, model_response_str -  experience.

**For Your Customers:**

* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.
* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337446, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=321, prompt_tokens=37, total_tokens=358, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337446, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=321, prompt_tokens=37, total_tokens=358, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337446, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' experience.\n\n**For Your Customers:**\n\n* **Misinformation & Misguidance:**  Inaccurate information due to prompt-hacking can mislead customers, leading them to make wrong decisions or take inappropriate actions based on faulty information.\n* **Frustration and Negative Experience:**  Unreliable and unhelpful responses due to manipulation', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337446, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': " will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 385, 'total_tokens': 422}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': " will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 385, 'total_tokens': 422}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': " will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 385, 'total_tokens': 422}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': " will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **"}
self.sent_first_chunk: True
hold - False, model_response_str -  will frustrate customers, damage their experience with your brand, and potentially drive them away.
* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.

**Mitigating the Risks:**

* **
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337447, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=385, prompt_tokens=37, total_tokens=422, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337447, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=385, prompt_tokens=37, total_tokens=422, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337447, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=" will frustrate customers, damage their experience with your brand, and potentially drive them away.\n* **Privacy and Security Concerns:**  Successful prompt-hacking could expose customers' personal information or make the entire system vulnerable to attacks, raising serious privacy and security concerns.\n\n**Mitigating the Risks:**\n\n* **", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337447, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': "Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 449, 'total_tokens': 486}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': "Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 449, 'total_tokens': 486}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': "Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could", 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 449, 'total_tokens': 486}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': "Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could"}
self.sent_first_chunk: True
hold - False, model_response_str - Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.
* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content="Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337449, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=449, prompt_tokens=37, total_tokens=486, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content="Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337449, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=449, prompt_tokens=37, total_tokens=486, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content="Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337449, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content="Robust Prompt Engineering:** Invest in robust prompt engineering techniques that anticipate and mitigate potential vulnerabilities. This includes thorough testing, adversarial training, and implementing safeguards against malicious inputs.\n* **Regular Monitoring & Auditing:** Regularly monitor the chatbot's interactions, analyze user inputs, and audit the system for any suspicious activities or anomalies that could", role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337449, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 513, 'total_tokens': 550}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 513, 'total_tokens': 550}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 513, 'total_tokens': 550}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they'}
self.sent_first_chunk: True
hold - False, model_response_str -  indicate prompt-hacking attempts.
* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.
* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337450, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=513, prompt_tokens=37, total_tokens=550, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337450, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=513, prompt_tokens=37, total_tokens=550, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337450, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' indicate prompt-hacking attempts.\n* **Rate Limiting & Input Validation:** Implement measures like rate limiting to restrict the number of requests from a single source and robust input validation techniques to filter out malicious or suspicious prompts.\n* **User Feedback & Reporting Mechanisms:** Encourage users to report any suspicious behavior or inaccurate information they', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337450, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 577, 'total_tokens': 614}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 577, 'total_tokens': 614}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 577, 'total_tokens': 614}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you'}
self.sent_first_chunk: True
hold - False, model_response_str -  encounter, enabling you to identify and address prompt-hacking attempts quickly.
* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.

By proactively addressing the risks of prompt-hacking, you
returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=577, prompt_tokens=37, total_tokens=614, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=577, prompt_tokens=37, total_tokens=614, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' encounter, enabling you to identify and address prompt-hacking attempts quickly.\n* **Security Best Practices:** Follow general security best practices for your customer portal, including data encryption, access control, and regular security audits to prevent unauthorized access and data breaches.\n\nBy proactively addressing the risks of prompt-hacking, you', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
value of async chunk: {'text': ' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 600, 'total_tokens': 637}, 'index': 0}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': ' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 600, 'total_tokens': 637}, 'index': 0}
model_response finish reason 3: None; response_obj={'text': ' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n', 'tool_use': None, 'is_finished': False, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 600, 'total_tokens': 637}, 'index': 0}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n'}
self.sent_first_chunk: True
hold - False, model_response_str -  can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system.

returning model_response: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=600, prompt_tokens=37, total_tokens=637, completion_tokens_details=None, prompt_tokens_details=None))
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None, usage=Usage(completion_tokens=600, prompt_tokens=37, total_tokens=637, completion_tokens_details=None, prompt_tokens_details=None))
Logging Details LiteLLM-Success Call: Cache_hit=False
final returned processed chunk: ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=' can leverage the power of AI-powered customer support while ensuring the reliability, accuracy, and security of your system. \n', role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
value of async chunk: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
PROCESSED ASYNC CHUNK PRE CHUNK CREATOR: {'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response finish reason 3: None; response_obj={'text': '', 'is_finished': False, 'finish_reason': '', 'usage': None, 'index': 0, 'tool_use': None}
model_response.choices[0].delta: Delta(content=None, role=None, function_call=None, tool_calls=None); completion_obj: {'content': ''}
self.sent_first_chunk: True
PROCESSED ASYNC CHUNK POST CHUNK CREATOR: None
Logging Details LiteLLM-Async Success Call, cache_hit=False
Logging Details LiteLLM-Success Call: Cache_hit=False
ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(content=None, role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
Goes into checking if chunk has hiddden created at param
Chunks have a created at hidden param
Chunks sorted
Looking up model=gemini/gemini-1.5-pro in model_cost_map
Logging Details LiteLLM-Async Success Call, cache_hit=False
Goes into checking if chunk has hiddden created at param
Chunks have a created at hidden param
Chunks sorted
Async success callbacks: Got a complete streaming response
Looking up model=gemini/gemini-1.5-pro in model_cost_map

Twitter / LinkedIn details

No response

bachya commented 1 month ago

A quick note: non-streaming calls (i.e., completion calls) always return a complete response.

krrishdholakia commented 3 weeks ago

value of async chunk: {'text': '', 'tool_use': None, 'is_finished': False, 'finish_reason': 'content_filter', 'usage': {'prompt_tokens': 37, 'completion_tokens': 17, 'total_tokens': 54}, 'index': 0}

@bachya it looks like this request is stopped mid-stream by gemini due to a content_filter error.

krrishdholakia commented 3 weeks ago

it looks like our final response doesn't return the correct stop_reason

ModelResponse(id='chatcmpl-044ed017-961c-4581-bdbe-2cd504b01dc4', choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(content=None, role=None, function_call=None, tool_calls=None), logprobs=None)], created=1728337451, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)