Open mneedham opened 1 month ago
@mneedham I need to add the stream on the model client, let me try to add it
@mneedham its updated, if you update pip to 0.1.0.b5
you should be able to stream the output
output = Generator()
for chunk in output.data:
print(chunk)
Awesome - it works :D Thanks!
I am testing it out with my usual ridiculous prompt!
model_client = OllamaClient(host="http://localhost:11434")
model_kwargs = {"model": "llama3.1", "stream": True}
generator = Generator(model_client=model_client, model_kwargs=model_kwargs)
output = generator({"input_str": "What would happen if a lion and an elephant met three dogs and four hyenas?"})
for chunk in output.data:
print(chunk, end='', flush=True)
What an interesting scenario!
If a lion and an elephant met three dogs and four hyenas, I think it's likely that the outcome would be quite dramatic.
Firstly, the lion would probably take charge of the situation, being the apex predator in the savannah. The elephants, being gentle giants, might try to stay calm and avoid any confrontation.
However, the presence of the three dogs could potentially cause a commotion. They might bark excitedly at the sight of the big cats, which could distract the lion and give the elephant an opportunity to intervene.
The four hyenas, on the other hand, would likely be more interested in scavenging for food than engaging in a full-blown battle. They might try to sneak up behind the dogs and steal their scraps, or even attempt to steal some of the elephant's food.
But if all else fails, I imagine the lion would assert its dominance by chasing after one of the smaller animals (perhaps the dogs?) to show who's boss. The elephant, being a gentle giant, might try to calm everyone down by using its size and presence to intimidate the hyenas into backing off.
Of course, this is all just hypothetical β in reality, each animal would behave according to their natural instincts and survival strategies! What do you think?
It's kinda neat that this code also works if I set stream to False because then .data
returns a string, which is also iterable :)
It's kinda neat that this code also works if I set stream to False because then
.data
returns a string, which is also iterable :)
Its a bug I introduced in the 0.1.0.b5. Please upgrade to .6 and it should work fine! (It werent supposed to change the normal non-stream behavior) π
Do we need some sort of await in parse_stream_response
to handle the acall
function?
output = await generator.acall({"input_str": "What would happen if a lion and an elephant met three dogs and four hyenas?"})
Error parsing the completion <async_generator object AsyncClient._stream.<locals>.inner at 0x10c9cac20>: argument of type 'async_generator' is not iterable
def parse_stream_response(completion: GeneratorType) -> Any:
"""Parse the completion to a str. We use the generate with prompt instead of chat with messages."""
for chunk in completion:
log.debug(f"Raw chunk: {chunk}")
yield chunk["response"] if "response" in chunk else None
It's kinda neat that this code also works if I set stream to False because then
.data
returns a string, which is also iterable :)Its a bug I introduced in the 0.1.0.b5. Please upgrade to .6 and it should work fine! (It werent supposed to change the normal non-stream behavior) π
I am using 0.1.0.b6!
It's kinda neat that this code also works if I set stream to False because then
.data
returns a string, which is also iterable :)Its a bug I introduced in the 0.1.0.b5. Please upgrade to .6 and it should work fine! (It werent supposed to change the normal non-stream behavior) π
I am using 0.1.0.b6!
Good, then there is no bug.
@liyin2015 does this function also need to check for AsyncGenerator
to have it work with the acall
function?
def parse_chat_completion(
self, completion: Union[GenerateResponse, GeneratorType]
) -> Any:
"""Parse the completion to a str. We use the generate with prompt instead of chat with messages."""
log.debug(f"completion: {completion}, {isinstance(completion, GeneratorType)}")
if isinstance(completion, GeneratorType): # streaming
return parse_stream_response(completion)
else:
return parse_generate_response(completion)
At the moment I get this error when using acall
with stream:True:
Error parsing the completion <async_generator object AsyncClient._stream.<locals>.inner at 0x124d731c0>: argument of type 'async_generator' is not iterable
@liyin2015 I tried a fix here, but I only did it for Ollama Client so far
Describe the bug
Not sure if this is a bug or if it's not supposed to work this way, but I can't figure out how to stream the response from the LLM.
To Reproduce
Returns:
Expected behavior
I want to be able to iterate over the response and render it as it's produced.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Mac OS Sonoma 14.5