search

nickna / Neighborly

An open-source vector database
MIT License
9 stars 2 forks source link

Potential race condition with MemoryMappedList ctor #62

Closed nickna closed 6 days ago

nickna commented 6 days ago

I'm investigating an issue where unit tests intermittently fail to access the physical files they've created. The tests pass when run individually. They fail or lock when run as a batch. This may be a Windows-related issue with file creation being inadvertently async.

image

CC: @hangy

nickna commented 6 days ago

stream = file. CreateViewStream() `is causing the exception: MemoryMappedFile System.IO.IOException: The paging file is too small for this operation to complete.

This is likely because the previous file(s) were not released before a new one was created. On Windows, MemoryMappedFile is limited based on virtual memory size, so it will not allocate more virtual memory than it has.

More to come...

hangy commented 6 days ago

One issue seems to be that VectorDatabase doesn't implement IDisposable. It should do so, so that VectorList gets disposed cleanly.

(This doesn't seem to remove all instances of this Exception on my PC, but it looks like it delays the first occurrence after a clean start)

hangy commented 6 days ago

It seems to me that with the current amount of memory allocated by the virtual files (albeit being sparse files) on Windows, they seem to break some limit if more than 2 instances of VectorDatabase are present at the same time. I was able to work around some of those issues with #63.

When running all tests, the system memory seems to be exhausted (I think it was running the Defrag tests at this point): image

nickna commented 6 days ago

That was quick. Your PR fixed the issue. Thank you!