How does the indexing works ?

[jeansordes]

Hi, I have 2000+ notes in my vault (400'000+ words), so I would like to understand how your code works for the indexing part before running it 😬 It might be worth my money … but I would like to be sure of that before commiting to it ( >ᴗ<)

[cpaika]

General indexing algorithm is:

• Split notes into sections
• batch sections, submit to OpenAI to get embeddings

We try to cram as many sections as possible into one request to minimize the amount of openai requests we are using. OpenAI has a free tier, I believe you should be able to index that vault within the free tier usage limits but I'm not certain. Let me know if you try it what happens

[windhamdavid]

@cpaika - Thank you for the work on this 👍🏼 I'm also curious as to the indexing and I've noticed that even in small-ish ( under 20 notes ) vaults, the "Your vault is still indexing!" message never disappears. I was just wondering if that's perhaps intentional or that my indexing isn't actually completing. I don't see any conditionals around the IndexingNotification but I figured I'd ask here before I started digging into the code.

// src/components/IndexingNotification.tsx
var import_jsx_runtime3 = __toESM(require_jsx_runtime());
var IndexingNotification = () => {
  return /* @__PURE__ */ (0, import_jsx_runtime3.jsx)("p", { className: "chat-indexing-banner", children: "Your vault is still indexing! Until indexing is complete, your chat assistant will only have partial context and results may be inaccurate. Indexing takes approximately 1 minute per 20 files in your vault." });
};