Open ned-pcs opened 1 month ago
I have tried uninstalling and reinstalling the extension.
Thanks @ned-pcs I will let the product team know now. How large is the repo (files and size)
Hey @ned-pcs sorry you're having these issues. If I see it correctly you're running on Linux right?
Could you have a look in the folder ~/.config/Code/User/workspaceStorage/sourcegraph.cody-ai
and specifically have a look if there's any .lock
files or folders in the symf
or cody-engine
directories? This is where VSCode will store these items it's trying to download.
If those lock
files are there we definitely need to figure out the root cause but in that case for now you can safely delete them and resetart VSCode to see if that works. If they aren't well keep digging to figure out what's going on. Are you running a firewall by any chance?
I'm on Ubuntu Linux 22.04 on an x86_64 system.
My ~/.config/Code/User/workspaceStorage
directory does not include a sourcegraph.cody-ai
directory. All of its directories have hex names. And none of those directories include a symf
or cody-engine
directory.
I did find those directories under ~/.config/Code/User/globalStorage/sourcegraph.cody-ai
:
% ls -a $(find ~/.config/Code/ -name cody-engine)
. .. bfg.zip cody-engine-5.4.6040-linux-x64.lock cody-engine-5.4.6040-linux-x64.tmp
% ls -a $(find ~/.config/Code/ -name symf)
. .. indexroot symf-v0.0.10-x86_64-linux symf-v0.0.12-x86_64-linux.lock symf-v0.0.12-x86_64-linux.tmp
After deleting the .tmp
and .lock
files and re-starting VS Code, I still see the same issue.
Ok, that's great. You are correct that it's the globalStorage
directory; I mixed it up when looking up the linux directory online (I'm on macOS). Thanks for taking the time to get those details regardless, it helps a lot!
The process for downloading looks like this:
.lock
So if we now first check if there's a zip file inside the .tmp
dir we can see if there's a problem with the downloading or the unpacking. Could you check this?
If there are zip files in there we can verify they're the same as if manually downloaded from https://github.com/sourcegraph/bfg/releases/download/v5.4.6040/bfg-linux-x64.zip
and https://github.com/sourcegraph/symf/releases/download/v0.0.12/symf-x86_64-linux-musl.zip
If they are, then we can verify if installing them manually resolves things. You can do this by unzipping the archive and copying the contents to the directory where the lockfiles are. Then you rename the binary to match the lockfile name (without .lock
), remove the lockfile and tmp dir, and restarting VSCode. If that works then there's some problem in unpacking these archives.
In the meantime I'll make sure to add some better error handling / reporting on these downloads
I also just noticed in the download code https://sourcegraph.com/github.com/sourcegraph/cody/-/blob/vscode/src/local-context/download-symf.ts?L113-171 that I added a different message for when it's extracting vs. downloading.
Given that the image you sent shows "downloading" I'm guessing that for whatever reason https://sourcegraph.com/github.com/sourcegraph/cody/-/blob/vscode/src/local-context/utils.ts?L38-49 isn't getting resolved.
I'm looking into why we're not receiving a finish
signal.
OK, after clearing out the cody-engine
and symf
dirs and restarting VS Code, I see no trace of bfg
in the cody-engine
dir. However, there are partial cody-engine
and symf
zip files there:
536768 Jul 25 09:03 ./cody-engine/cody-engine-5.4.6040-linux-x64.tmp/cody-engine-5.4.6040-linux-x64.zip
278201 Jul 25 09:03 ./symf/symf-v0.0.12-x86_64-linux.tmp/symf-v0.0.12-x86_64-linux.zip
I don't have the download link for the cody-engine
zip file; I could try unpacking that plus the bfg
zip in the cody-engine
dir, and the symf
zip in the symf
dir if that's what is needed.
After cleaning out those directories and re-starting in a much smaller repo, I see both executables installed:
/home/ned/.config/Code/User/globalStorage/sourcegraph.cody-ai:
total 8
drwxrwxr-x 2 ned ned 4096 Jul 26 09:24 cody-engine
drwxrwxr-x 3 ned ned 4096 Jul 26 09:31 symf
/home/ned/.config/Code/User/globalStorage/sourcegraph.cody-ai/cody-engine:
total 193152
-rwxr-xr-x 1 ned ned 197779648 Jul 26 09:24 cody-engine-5.4.6040-linux-x64
/home/ned/.config/Code/User/globalStorage/sourcegraph.cody-ai/symf:
total 31148
drwxrwxr-x 4 ned ned 4096 Jul 26 09:31 indexroot
-rwxr-xr-x 1 ned ned 31889320 Jul 26 09:31 symf-v0.0.12-x86_64-linux
However, looking at the logs, I wonder whether those zips are being downloaded and installed every time I start up the extension, despite the fact that they're already the latest version.
I will now start VS Code on my original repository.
Thanks @ned-pcs I will let the product team know now. How large is the repo (files and size)
This is a repo containing submodules; most of the files are not of interest, but overall there are:
53290 C files
4369 Python files
2173 Javascript files
as well as a number of miscellaneous binary blobs, documents, and other things.
I don't see how to limit the scope of the search.
It appears that indexing fails for this repository.
Here's the output log for the larger repo:
telemetry-v2: recordEvent: cody.extension/savedLogin
█ AuthProvider:init:lastEndpoint: Token recovered from secretStorage https://sourcegraph.com/
█ GraphQLTelemetryExporter: evaluated export mode: 5.2.5+
█ CodyLLMConfiguration: {"chatModel":"anthropic/claude-3-sonnet-20240229","chatModelMaxTokens":12000,"fastChatModel":"anthropic/claude-3-haiku-20240307","fastChatModelMaxTokens":12000,"completionModel":"fireworks/starcoder","completionModelMaxTokens":9000,"provider":"various","smartContextWindow":true}
█ ClientConfigSingleton: refreshing configuration
█ ClientConfigSingleton: refreshed {"codyEnabled":true,"chatEnabled":true,"autoCompleteEnabled":true,"customCommandsEnabled":true,"attributionEnabled":false,"smartContextWindowEnabled":true,"modelsAPIEnabled":false}
█ ModelsService: Setting primary models: ["anthropic/claude-3-5-sonnet-20240620","anthropic/claude-3-sonnet-20240229","anthropic/claude-3-opus-20240229","anthropic/claude-3-haiku-20240307","openai/gpt-4o","openai/gpt-4-turbo","openai/gpt-3.5-turbo","google/gemini-1.5-pro-latest","google/gemini-1.5-flash-latest","fireworks/accounts/fireworks/models/mixtral-8x7b-instruct","fireworks/accounts/fireworks/models/mixtral-8x22b-instruct"]
█ telemetry-v2: recordEvent: cody.auth/connected
█ featureflag: refreshed
█ ContextFiltersProvider: setContextFilters
█ SymfRunner: unsafeEnsureIndex file:///home/ned/PCS/Pyrenees/Chihuahua-Firmware
█ LocalEmbeddingsController: constructor
█ ChatsController:constructor: init
█ LocalEmbeddingsController: start
█ CodyEngineService: constructor
█ LocalEmbeddingsController: start
█ openctx: OpenCtx is enabled in Cody
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"indeterminate","embeddingsAPIProvider":"sourcegraph"}]}]
█ CodyEngine: using downloaded bfg path "/home/ned/.config/Code/User/globalStorage/sourcegraph.cody-ai/cody-engine/cody-engine-5.4.6040-linux-x64"
█ CodyEngineService: spawnAndBindService service started, initializing
█ LocalEmbeddingsController: spawnAndBindService service started, initializing
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"indeterminate","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ CodyEngine: spawnBfg:stderr
█ symf: using downloaded symf "/home/ned/.config/Code/User/globalStorage/sourcegraph.cody-ai/symf/symf-v0.0.12-x86_64-linux"
█ LocalEmbeddingsController: spawnAndBindService initialized
█ CodyCompletionProvider:initialized: fireworks/starcoder-hybrid
█ telemetry-v2: recordEvent: cody.context.embeddings/loaded
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"unconsented","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ LocalEmbeddingsController: index starting repository file:///home/ned/PCS/Pyrenees/Chihuahua-Firmware
█ telemetry-v2: recordEvent: cody.context.embeddings/loaded
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"unconsented","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ LocalEmbeddingsController: index starting repository file:///home/ned/PCS/Pyrenees/Chihuahua-Firmware
█ LocalEmbeddingsController: index-health c2e61bd83c05478fb56bc3639d5ad062 {"code":-32099}
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"indexing","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ UpstreamHealth: Ping took 4065ms (Gateway: 4026ms)
█ LocalEmbeddingsController: index-health 568a40cfb1734c308fcfe82e6763cbc1 {"code":-32099}
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"indexing","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ symf: creating index file:///home/ned/.config/Code/User/globalStorage/sourcegraph.cody-ai/symf/indexroot/home/ned/PCS/Pyrenees/Chihuahua-Firmware
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"unconsented","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ telemetry-v2: recordEvent: cody.context.embeddings/loaded
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"ready","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ LocalEmbeddingsController: load after indexing "done" file:///home/ned/PCS/Pyrenees/Chihuahua-Firmware true
█ LocalEmbeddingsController: index-health {"commit":"7c4c5a5b1905179ae817cb28f4d939af8cfe06a7","dimension":768,"format":"LocalEmbeddings","model":"sourcegraph/st-multi-qa-mpnet-base-dot-v1","numFiles":221,"numItems":1197,"numItemsDeleted":0,"numItemsFailed":0,"numItemsNeedEmbedding":0,"repoName":"github.com/Product-Creation-Studio/Pyrenees-NFF-Firmware","type":"found"}
█ LocalEmbeddingsController: get status
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"ready","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"indexing"}]}]
█ UpstreamHealth: Ping took 159ms (Gateway: 150ms)
█ symf: symf index creation failed EvalError: symf index creation failed: Error: symf exited with code null
█ ChatController: postContextStatus [{"displayName":"Chihuahua-Firmware","providers":[{"kind":"embeddings","state":"ready","embeddingsAPIProvider":"sourcegraph"},{"kind":"search","type":"local","state":"failed"}]}]
When I look at the contents under symf/indexroot
I see that it only made its way into one directory within a submodule. This submodule's origin is in a public Github repository.
If it matters, the main repository's origin is a private repository on Github.
Ok I think the indexing is separate from the downloading / installation issue. I'll create a separate internal issue for it so we can get the right people looking at that one.
For the downloading; I think the issue might be that for some reason the previous request might have gotten interrupted and there were partial files left that prevented writing. This could maybe happen if VSCode restarted right as it was downloading. Alternatively...did you at any point start VSCode as sudo
? I'll need to dig a little deeper to form a bit more solid theory.
The cody-engine is this file https://github.com/sourcegraph/symf/releases/download/v0.0.12/symf-x86_64-linux-musl.zip , it's just named differently. And although it will check for the latest versions every time you open Cody it won't re-download it if it's already available.
Version
v1.26.7
Describe the bug
Every time I start up VS Code, I see notifications that Cody is downloading![Screenshot from 2024-07-24 11-40-43](https://github.com/user-attachments/assets/9412d518-3d31-48ca-860a-fc04696179bf)
bfg
andsymf
, and that it is updating the search index. However, these notifications never go away.Expected behavior
I expect the downloads to complete and the search index to be updated.
Additional context
Here's the output from Cody: