search

riatzukiza / duck

0 stars 0 forks source link

Write a class to encapsulate indexed mongodb, file systems, apis, etc. #5

Open riatzukiza opened 1 week ago

riatzukiza commented 1 week ago

To help clean up the code, somthing that we do a lot is index collections of stuff, whether that is a local file system, a mongodb collection of discord messages, or a table of webpages scraped from search results.

Idenitify actions taken in current code that cross correlate data stored in a primary source, that is indexed in a vector database.

Ideally the solution is one that scales over many data sources, and possible choices for vector databases.

riatzukiza commented 1 week ago

We started this yesterday, will post wip pr link soon.