Open Jozurf opened 2 hours ago
On the topic of KvBufferCache
, is there a limit to how big the kvBufferCache
can grow? It creates a new entry in its map each time AbstractModel.embed is called based on current implementation and I cant find a place where we are removing entries. If there exists an application that calls AbstractModel.embed() a large number of times (e.g. 1 billion), then we might cause a slowdown because of the number of entries the kvBufferCache has. Is there a quick solution for that?
Problem
When loading an embedding model, I noticed that calling
model.embed
multiple times eventually results in ajava.io.FileNotFoundException
due to "too many open files." Theembed
method inAbstractModel
generates a newrandomUUID
as a session ID for each call, creating a new entry inkvBufferCache
with a newKvBuffer
. While eachKvBuffer
instance is created using a try-with-resources statement, theclose()
method of theKvBuffer
class is empty. This means resources allocated within thatKvBuffer
in theKvBufferCache
are not released. EachKvBuffer
contains a 2D array ofKvBufferPage
, and sinceKvBufferPage
is created (specifically atgetTensorForPosition
inKvBuffer
) during the embedding but not closed, eachKvBufferPage
holds on to aRandomAccessFile
and is never released. Here is the stack trace that I encountered.Solution
I suggest closing each KvBufferPage in the close() method of KvBuffer.