teamblubee
commented
2 months ago I found the issue in the way the chat_completion function is defined, it's printing the error in the stream....
async def chat_completion(self, request: ChatCompletionRequest) -> AsyncGenerator:
async with httpx.AsyncClient() as client:
async with client.stream(
"POST",
f"{self.base_url}/inference/chat_completion",
json={
"request": encodable_dict(request),
},
headers={"Content-Type": "application/json"},
timeout=20,
) as response:
if response.status_code != 200:
content = await response.aread()
cprint(
f"Error: HTTP {response.status_code} {content.decode()}", "red"
)
return
async for line in response.aiter_lines():
if line.startswith("data:"):
data = line[len("data: ") :]
try:
# Debugging: print out the structure of the data
# print(f"Raw response: {data}")
if request.stream:
if "error" in data:
print(f"ERROR IN STREAM")
cprint(data, "red") #<------------------------------- THIS LINE
continue
# Yielding the streamed chunk response
yield ChatCompletionResponseStreamChunk(**json.loads(data))
else:
# Yielding the complete response at once
yield ChatCompletionResponse(**data)
except Exception as e:
print(data)
print(f"Error with parsing or validation: {e}")
Testing on llama 3.1 8B, local version
Don't mind this sloppy code, I'm trying to understand the response format since it seems the documentation has changed.
I keep getting json looking things like these in the response and cannot get them out via code.
What is that event and how can I properly parse it from the response?
response
Chemical Properties
Chemical properties are characteristics of a substance that can be observed or measured. Some common chemical properties include:
Chemical Reactions and Equations
Chemical reactions and equations are used to describe the transformation of one or more substances into new substances. Some common types of chemical reactions include:
References
Age of Chemistry
Chemistry has a long and rich history that dates back to ancient times. Some notable milestones in the development of chemistry include:
Years of Study
The study of chemistry typically takes several years to complete. Here is a rough outline of the typical progression of a chemistry student:
Note: The exact progression may vary depending on the individual and their goals.
Chemical Properties
Chemical properties are characteristics of a substance that can be observed or measured. Some common chemical properties include:
Chemical Reactions and Equations
Chemical reactions and equations are used to describe the transformation of one or more substances into new substances. Some common types of chemical reactions include:
References
Age of Chemistry
Chemistry has a long and rich history that dates back to ancient times. Some notable milestones in the development of chemistry include:
Years of Study
The study of chemistry typically takes several years to complete. Here is a rough outline of the typical progression of a chemistry student:
Note: The exact progression may vary depending on the individual and their goals.