Open isConic opened 6 months ago
Hey, yes, we are working on this feature, (we welcome contributions as well!) and our approach is quite similar to the one you have described!
I might be interested in contributing but would like to make sure any contribution wouldn't conflict/contradict your approachto this or your vision. What exactly do you have in mind for add_to_existing
?
@parthsarthi03 Any updates on this?
Any updates on this functionality?
https://github.com/parthsarthi03/raptor/blob/9d99cbda1b2cb043dde22acc9ef8c4b0bc642392/raptor/RetrievalAugmentation.py#L216.
Problem. If I want to iteratively add documents to a raptor index. Whenever I call
add_documents
I am met with an un-avoidable message:For the 200 or so documents that I wish to index, typing
y
each time is not feasible. Especially if I attempt to scale to thousands or tens of thousands in the future.Barring the inconvenience of there being no way to override to this, upon inspecting the code, I've realized that selecting
yes
ory
just terminates the function without doing anything. In the code block that's responsible for handling theyes
there is this line that's commented out, followed by a return statement:Wishing to still iteratively add to the index, I've searched the codebase coming to the realization that the
add_to_existing
method does not actually exist anywhere in the codebase.Speculation:.
Perhaps the intended approach to
add_to_existing
is too difficult. The approach may have initially been to build one large tree for all documents and then rebalance already constructed trees with the addition of new documents?If avoiding solving that monolith tree-rebalancing problem is why
add_to_exisitng
does not exist, I speculate that if raptor instead maintained an index of several smaller disjointed trees, it would be more appropriate of an index as that allows for scaling. The entry point to a tree not being the top or root of a single tree, but any node that is semantically close to the question from an embedding space encompassing multiple trees (every file that's indexed gets a tree).The algorithm would pull in context in both directions (up the hierarchy and down the hierarchy).