Local semantic search over folders. Why didn't this exist?
pip install aifs
pip install "unstructured[all-docs]" # If you want to parse all doc types. Includes large packages!
from aifs import search
search("How does AI Filesystem work?", path="/path/to/folder")
search("It's not unlike how Spotlight works.") # Path defaults to CWD
Running aifs.search
will chunk and embed all nested supported files (.txt
, .py
, .sh
, .docx
, .pptx
, .jpg
, .png
, .eml
, .html
, and .pdf
) in path
. It will then store these embeddings into an _.aifs
file in path
.
By storing the index, you only have to chunk/embed once. This makes semantic search very fast after the first time you search a path.
If a file has changed or been added, aifs.search
will update or add those chunks. We still need to handle file deletions (we welcome PRs).
unstructured
to parse and chunk every file in the path
.chroma
to embed the chunks locally and save them to a _.aifs
file in path
.chroma
is used again to semantically search the embeddings.If an _.aifs
file is found in a directory, it uses that instead of indexing it again. If some files have been updated, it will re-index those.
python
and markdown
files should have different chunking algorithms based on the expected content of those filetypes. Who has this solution?aifs
to be the best local semantic search in the universe.We built this to let open-interpreter
quickly semantically search files/folders.