Now that streaming works in text gen v2, enable streaming in the OpenAI server
Testing
Starting the server with --integration openai, we can send the python api request below
import openai
from openai import OpenAI
client = OpenAI(base_url="http://localhost:5543/v1", api_key="EMPTY")
models = client.models.list()
model = "hf:mgoin/TinyStories-1M-ds"
print(f"Accessing model API '{model}'")
# Completion API
stream = True
completion = client.chat.completions.create(
messages="The sun shined",
stream=stream,
max_tokens=10,
model=model
)
for c in completion:
print(c)
Summary
Testing
Starting the server with
--integration openai
, we can send the python api request belowOutput: