Symbol cache staleness when using a `develop`ed package

[tpgillam]

This issue describes behaviour observed in neovim and LanguageServer.jl, via julia-lsp. Having dug around in the code a bit, it looks like the caching logic is here rather than LanguageServer.jl, hence filing here.

Case 1 (the issue)

Consider the following file structure within some directory.

├── A
│   ├── Project.toml
│   └── src
│       └── A.jl
└── proj
    ├── Manifest.toml
    ├── Project.toml
    └── test.jl

There is a Julia package called "A", and beside it is a project called "proj". A is a package generated with import Pkg; Pkg.generate("A") from the top-level directory.

As-of Julia 1.10.1, there is a single A.greet() method defined.

Within proj we run ] dev ../A, such that its Manifest.toml contains:

[[deps.A]]
path = "../A"
uuid = "94f385dd-073b-49fe-b7ed-f824d09b3331"
version = "0.1.0"

Within proj/test.jl we write:

using A

A.greet()

And the language server can find A and A.greet, so all is well.

Now, close add a new function into A/src/A.jl, e.g.

moo() = 42

Back in proj/test.jl, now extend the file to call the new function. The language server now declares the new symbol as missing:

using A

A.greet()
A.moo()    # Missing reference: moo

Restarting the language server (or the whole editor) doesn't change anything.

We can see that when the language server starts, it loads A from the cache, but doesn't detect that the cache is stale:

"[ Info: Starting LS at 1708163760\n"
"[ Info: Package A (94f385dd-073b-49fe-b7ed-f824d09b3331) is cached.\n"
"[ Info: Symbol server indexing took 5.283324494 seconds.\n"
"[ Info: Loading A from cache... (0%)\n"
"[ Info: Done loading A from cache... (took 0.12s) (0%)\n"
"[ Info: Loaded all packages into cache in 0.12s (100%)\n"

Case 2 (similar, but works)

Now we consider a slightly different case, where A is now underneath the project directory, as follows:

├── A
│   ├── Project.toml
│   └── src
│       └── A.jl
├── Manifest.toml
├── Project.toml
└── test.jl

In this situation the issue above is not seen. Instead, if we edit A/src.jl and then open test.jl, when the language server starts we see:

"[ Info: Starting the Julia Language Server\n"
"[ Info: Symbol server store is at '/home/tom/.local/share/nvim/mason/packages/julia-lsp/libexec/store/symbolstorev5
"[ Info: Starting LS at 1708164676\n"
"[ Info: Outdated sha, will recache package A (94f385dd-073b-49fe-b7ed-f824d09b3331)\n"
"[ Info: Indexing A... (0%)\n"
"[ Info: Now writing to disc 94f385dd-073b-49fe-b7ed-f824d09b3331\n"
"[ Info: Symbol server indexing took 5.497283574 seconds.\n"
"[ Info: Loading A from cache... (0%)\n"
"[ Info: Done loading A from cache... (took 0.067s) (0%)\n[ Info: Loaded all packages into cache in 0.067s (100%)\n"

where this time it detects that A has changed and needs to be updated.

So I think the solution to this issue will involve ensuring that we look for changes in all developed packages in the Manifest, regardless of their relative path.

---

A little poking around suggests to me that the language server is caching its view of symbols in packages in .jstore files. In my case, the storage for the A package ends up here:

~/.local/share/nvim/mason/packages/julia-lsp/libexec/store/symbolstorev5/A/A_94f385dd-073b-49fe-b7ed-f824d09b3331

Manually deleting this directory, and restarting the language server, means that the cache will get rebuilt, and we see any new symbols.