microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps
https://aka.ms/semantic-kernel
MIT License
20.92k stars 3.06k forks source link

.Net TextMemoryPlugin sample15 doesn't work with new openai models #5346

Open Krzysztof318 opened 5 months ago

Krzysztof318 commented 5 months ago

Describe the bug A clear and concise description of what the bug is.

TextMemoryPlugin doesn't work correctly with new models: text-embedding-3-small and text-embedding-3-large

To Reproduce Steps to reproduce the behavior: Run sample15 (volataile)

Output

== PART 1a: Saving Memories through the ISemanticTextMemory object ==
Saving memory with key 'info1': "My name is Andrea"
Saving memory with key 'info2': "I work as a tourist operator"
Saving memory with key 'info3': "I've been living in Seattle since 2005"
Saving memory with key 'info4': "I visited France and Italy five times since 2015"
== PART 1b: Retrieving Memories through the ISemanticTextMemory object ==
Memory with key 'info1':My name is Andrea

== PART 2a: Saving Memories through the Kernel with TextMemoryPlugin and the 'Save' function ==
Saving memory with key 'info5': "My family is from New York"
== PART 2b: Retrieving Memories through the Kernel with TextMemoryPlugin and the 'Retrieve' function ==
Memory with key 'info5':My family is from New York

== PART 3: Recall (similarity search) with AI Embeddings ==
== PART 3a: Recall (similarity search) with ISemanticTextMemory ==
Ask: where did I grow up?
== PART 3b: Recall (similarity search) with Kernel and TextMemoryPlugin 'Recall' function ==
Ask: where do I live?
Answer: 

== PART 4: Using TextMemoryPlugin 'Recall' function in a Prompt Function ==
Ask: Do I live in the same town where I grew up?
Answer: Based on the provided facts, there is no information about where you grew up or currently live. Therefore, I cannot determine if you live in the same town where you grew up.
== PART 5: Cleanup, deleting database collection ==
Printing Collections in DB...
aboutMe

Removing Collection aboutMe

Printing Collections in DB (after removing aboutMe)...

Platform

Additional context Add any other context about the problem here.

Nurgo commented 4 months ago

Indeed, the new OpenAI embedding models have a completely different distribution of their results, minRelevanceScore should be set to something like 0.3 instead of 0.7 (current default value), otherwise in most cases no results will be returned.

github-actions[bot] commented 1 month ago

This issue is stale because it has been open for 90 days with no activity.

dmytrostruk commented 1 month ago

Bumping this issue as it still exists.