deepset-ai / haystack-core-integrations

Additional packages (components, document stores and the likes) to extend the capabilities of Haystack version 2.0 and onwards
https://haystack.deepset.ai
Apache License 2.0
121 stars 119 forks source link

feat: add `keep_alive` parameter to Ollama Generators #1131

Closed emso-c closed 1 month ago

emso-c commented 1 month ago

Related Issues

Proposed Changes:

Added keep_alive parameter to OllamaGenerator and OllamaChatGenerator

How did you test it?

Ran unit tests:

hatch run test -m"not integration"

Manual tests: Set keep alive to 0 and checked process consumptions.

from haystack_integrations.components.generators.ollama import OllamaGenerator, OllamaChatGenerator
from haystack.dataclasses import ChatMessage, ChatRole

generator = OllamaGenerator(keep_alive=0)
replies = generator.run("Say hello").get("replies")
assert replies and replies[0].strip() != ''

generator = OllamaChatGenerator(keep_alive=0)
message = ChatMessage(content="Hello", role=ChatRole.USER, name=None)
replies = generator.run([message]).get("replies")
assert replies and replies[0].meta['done'] == True

Expected behavior:

The processes immediately getting unloaded from memory.

Actual behaviour:

The processes immediately getting unloaded from memory.

Notes for the reviewer

Checklist

emso-c commented 1 month ago

@anakin87 now that we addressed that the default value is "5m" in the docs, you think we should update the default value to "5m" in the code as well? It might be confusing for users to see that the value is None, contradicting the docs.