search

microsoft / kernel-memory

RAG architecture: index and query any data using LLM and natural language, track sources, show citations, asynchronous memory patterns.
https://microsoft.github.io/kernel-memory
MIT License
1.52k stars 293 forks source link

[Bug] Content Storage File handles not properly disposed #475

Closed swimmesberger closed 4 months ago

swimmesberger commented 4 months ago

Context / Scenario

When using Kernel Memory with the In-Process Orchestrator and SimpleFileStorage (Volatile) I'm encountering issues when adding documents. I looked into the code and was wondering if the StreamableFileContent objects returned from the IContentStorage are not disposed in some scenarios.

https://github.com/microsoft/kernel-memory/blob/6b0d3477389c17d8efc99bda60610f8562b94c51/service/Core/Pipeline/BaseOrchestrator.cs#L156 https://github.com/microsoft/kernel-memory/blob/6b0d3477389c17d8efc99bda60610f8562b94c51/service/Core/Pipeline/BaseOrchestrator.cs#L243

What happened?

 System.IO.IOException: The process cannot access the file 'C:\...\km-files\default\doc.docx\__pipeline_status.json' because it is being used by another process.
         at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
         at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
         at System.IO.File.OpenHandle(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
         at System.IO.File.<WriteAllBytesAsync>g__Core|92_0(String path, Byte[] bytes, CancellationToken cancellationToken)
         at Microsoft.KernelMemory.FileSystem.DevTools.DiskFileSystem.WriteFileAsync(String volume, String relPath, String fileName, Stream streamContent, CancellationToken cancellationToken)       
         at Microsoft.KernelMemory.ContentStorage.DevTools.SimpleFileStorage.WriteFileAsync(String index, String documentId, String fileName, Stream streamContent, CancellationToken cancellationToken)
         at Microsoft.KernelMemory.Pipeline.BaseOrchestrator.UpdatePipelineStatusAsync(DataPipeline pipeline, CancellationToken cancellationToken)

Importance

I cannot use Kernel Memory

Platform, Language, Versions

Platform: Windows 11 Pro 22631 Kernel Memory Version: 0.50.240504.7 .NET Version: 8.0.300-preview.24203.14 Language: C#

Relevant log output

No response

marcominerva commented 4 months ago

I was able to reproduce the issue. I have added the using keyword when working with StreambleFileContent, now the problem should be fixed.

dluc commented 4 months ago

Fixed, thanks @marcominerva :-)