Configuration value `disableIndexing` appears to have no effect

[andrewbrey]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:Linux (Pop_OS! 22.04)
- Continue: v0.9.28 (pre-release version)
- IDE: VSCode 1.85.2

Description

I have set "disableIndexing": true in my ~/.continue/config.json however the extension still continues to index my project files:

A quick search in the repo for disableIndexing seems to indicate that this value does not influence any behavior as it is only referenced in types and schema files:

Is this a configuration value that is deprecated? Is it unimplemented? Thank you for any clarification you can provide!

To reproduce

No response

Log output

No response

[sestinj]

@andrewbrey This is just un-implemented since we changed some of our indexing code. But it shouldn't be! Look for it to work in the next pre-release version today or tomorrow

[sestinj]

I'd also like to understand some of the reasons that people choose to disable, because the ideal scenario is that there's some better default. Is it largely due to high CPU use?

[andrewbrey]

@sestinj Thank you, that is great to hear!

My reasons for disabling are two-fold:

1. My personal use case for in-editor LLMs is quick access to question and answer about a topic rather than actual code completion. To me it acts as a much better default search engine that lets me not have to change context to look something up. This does not require any context from my project files.
2. The indexing is very slow for large projects (like the kind I work on) and I have noticed that it has a non-trivial impact on the responsiveness of my editor.

[sestinj]

thanks! this is good to know. I'll update here when the new version is released

[andrewbrey]

@sestinj FWIW, I would be more inclined to keep it enabled if:

(a) I could have more fine-grained control over which files it indexed, in particular I wish that at the least it respected my .gitignore paths in choosing files to skip for indexing. (b) I could have easy access to the indexing service interface to inspect what has been indexed to help build trust - meili search is being used right? Can the server be linked to from the extension?

[sestinj]

@andrewbrey It should respect both .gitignore and .continueignore, but I'll double check this. Is a very important point, and implementation is a bit undertested since it's so new

The entire Python server, which meilisearch is a part of, is no longer included. Continue now runs entirely self-contained. The index is stored at ~/.continue/index, where embeddings are in the lancedb folder, but the overall indexing metadata is stored in index.sqlite. The best way to see what's indexed is probably to download an application like https://sqlitebrowser.org/ and browse through the tags_catalog table of that file

Both really good to know, both actionable. For the latter a bit of explanation in the docs would be useful for us to add

[andrewbrey]

@sestinj This is incredibly helpful, thank you and I appreciate you mentioning that it would be good to add to docs, and thank you for the clarification on the inspectability of the search indices, I will poke around in there.

With respect to the .gitignore, I will be very curious to see what a double check shows because when I hover over the "indexing progress bar" the hover text has (I'm like 98% sure) shown files which are part of my .gitignore.

Thank you very much for Continue and for indulging my slightly off-topic extended discussions in this issue :)

[sestinj]

@andrewbrey this was a quick one: https://github.com/continuedev/continue/blob/88e81221f08f25ea5fcdb379541055574691f49f/extensions/vscode/src/util/indexCodebase.ts#L40-L43

Solved now in pre-release version 0.9.29

[sestinj]

Added a small note about viewing the sqlite table to the docs, so now closing this issue