search

microsoft / kernel-memory

RAG architecture: index and query any data using LLM and natural language, track sources, show citations, asynchronous memory patterns.
https://microsoft.github.io/kernel-memory
MIT License
1.52k stars 293 forks source link

[Question] Running Kernel Memory as a service in Asp.Net #493

Closed Robban1980 closed 4 months ago

Robban1980 commented 4 months ago

Context / Scenario

I am looking at running Kernel Memory as a service in Asp.Net API.

Question

I took a look at the Service.AspNetCore code and saw that IKernelMemory is registered as a singleton. With an API i was wondering why we should register IKernelMemory as a singleton instead of as a scoped service?

dluc commented 4 months ago

The recommended deployment uses asynchronous queues to execute tasks in the background, to avoid blocking user requests and to deal with long running jobs.

For example, when uploading a file it might take seconds/minutes to process, depending on the pipeline in use, depending on external services bandwidth, and in case of transient errors requiring the service to retry.

To avoid blocking TCP/HTTP requests while documents are processed, the ingestion is designed to run as a task outside the scope of a HTTP request. Using a singleton allows KM components to work. It's also perfectly safe to use KM as a singleton.

On the other hand, if for some reason you cannot use singletons, then I would recommend configuring KM in Serverless mode. Everything will run synchronously, blocking HTTP requests while ingestion is running, and once a request is complete the KM instance can be disposed.