Issues with bloop cache and Metals

[szymon-rd]

While working with stack a consisting of Fury, Metals, bloop and VSCode I would encounter issues that mainly manifested as:

• Lack of or outdated code completion suggestions - edit: Sometimes restart of metals server would help. It occurred after working on a project for some time, and most commonly it couldn't provide information about libraries used in the project. It was the most common state - not having information about the libraries, but being able to see the project context.
• Being unable to build a new version of code base - each build would result in compilation of the old, cached one edit: This building issue occurred pretty rarely - about once in 10 builds I think, and then persisted until I restarted fury and deleted all the building-related dirs (.fury, .bloop, .metals). I will add this to the problem description. But important part I think is that deleting .bloop file didn't help. At this point I often tried restarting fury and deleting .metals and .fury file and I think that was the operation that made it work.

I'm using MacBook Pro with macOS 10.15.4 (19E287), accessing Fury from zsh.

[propensive]

Can you try something to diagnose this? I wonder if the caching issue might be a different fixed issue.

Are you able to make a deliberate breaking change to a file, attempt to recompile it, and appear to see successful compilation, reliably and repeatably?

Can you find the JSON config file in the .bloop directory for the module you're compiling, and look at the "sources" field. Is it pointing to the correct sources, or to a file in Fury's managed space?

[szymon-rd]

Is it pointing to the correct sources, or to a file in Fury's managed space?

It is pointing to correct sources in my workspace.

This building issue occurred pretty rarely - about once in 10 builds I think, and then persisted until I restarted fury and deleted all the building-related dirs (.fury, .bloop, .metals). I will add this to the problem description. But important part I think is that deleting .bloop file didn't help. At this point I often tried restarting fury and deleting .metals and .fury file and I think that was the operation that made it work.

The code suggestion issue I was never able to fix reliably. Sometimes restart of metals server would help. It occurred after working on a project for some time, and most commonly it couldn't provide information about libraries used in the project. It was the most common state - not having information about the libraries, but being able to see the project context.

[szymon-rd]

I think I reproduced the building cache issue. I had a module named webapp with classes structured within directory src/acrimony/. Then, I removed the source set from the module: fury source remove -d sz:src/acrimony Where sz is my repository. Then I created a new folder a/workaround and added this as a source. It resulted in updating the entry in bloop: "sources":["[some files...]/a/workaround"] Then, when I try to build this file: fury build run -F -d . It results in jar that contains both the new classes (from a/workaround/) and old ones (from acrimony/):

jar -tf webapp.jar | grep acrimony
acrimony/AccessDeniedException$.class
... and many more

Then, after removing .fury directory I run these commands again and there were no entries from acrimony dirs - so that fixed the issue. That is not exactly what happend in my previous problems - back then it just used the old classes. But I think this may be related.

[propensive]

This sounds similar to something I've also seen. Bloop, to my knowledge, does not delete .class files in the output directory, it only ever adds to them. So any old classes remain until a clean takes place. I think we can create a test which shows this, though I don't know how easy it would be to fix, as we don't want to force a full recompilation... The problem should exist with Metals too, however. I would be interested to see if it doesn't.

[propensive]

It will be interesting to see if fixing #1380 helps.

[veyga]

The class files are cached in a few places. Bloop outputs into analysis and classes directories. Looks like the analysis directory is the bloop cache with respect to the file system, although it has its in memory caches. The documentation says clearing the bloop cache clears memory caches, but any desired changes to the filesystem should be implemented by the client. It would appear that the classes directory is kind of 'ours' as a client to use as we please. So yes, every class file will persist there until its explicitly cleaned. Building a jar pulls all classes in the build target path and its dependencies into this 'staging' directory. Even after a clean, this staging directory persists, which is why the old classes were still packaged with the jar.

If you delete the staging directory after each jar creation, then you'll have a correct jar assuming you 'fury clean classes' beforehand. How to delete old class files without cleaning everything is the question. I think the easiest thing would be to clean an individual module after a set of sources is removed. This way you're not having to recompile everything, whilst ensuring a clean build. Another option would be to introduce some sort of blacklist, so that entities are added to a blacklist if you remove them as sources from a given module, which would get hairy though as an implementation.

Its really hard to track whats old, since the interface between fury and bloop deals at a module/directory level, and doesn't appear to have any mechanism for tracking down individual class files. Even worse if you're creating/reestablishing the connection, you'd be forced determine what to clean specifically just by digging through the raw class files.

As to how specifically this is messing with metal's ability, I don't know, but it seems like its referencing the analysis directories, and not the (potentially dirty) classes directory. So it shouldn't be an issue, unless you're working with jars that fury created.

TLDR. Looks like this is client issue to solve, and not something that bloop itself would address, but I could be wrong.