Initializing plugin in a mid-sized monorepo takes hours

[tomasaschan]

We have a monorepo with a couple of hundred subcomponents (as determined by find . -name BUILD.bazel in the monorepo root). Opening a subpath of that monorepo and initializing the plugin has been running for hours and I have no idea how long it's going to take until it completes.

A few steps I've noticed took a lot of time:

• Provisioning .eclipse projects for each java component
• Computing Bazel project classpaths for a bunch of stuff; unclear to me what the scope of each of those is, because sometimes it seems like it's doing it for an entire package, and other times for just a single file.

At the step I'm currently at, every so often (a few times per minute?) the Java Build Status view outputs something like {hash} {repo-name}> bazel query //{path-to/subcomponent}:all [Done].

Is it expected that this extension doesn't work well with monorepos? Is it expected that it should take this long? (I thought monorepos was why you used Bazel, so that would surprise me...) How much of this work will be cached, and how much will have to happen every time I open a component in this monorepo? Will it matter if I've opened that exact component before or not?

[guw]

Yikes, it's likely going to anayze the entire mono-repo. There are a few performance issues with JDTLS that we don't see in Eclipse.

When opened inside a workspace, the extension tries to walk up the tree to discover the workspace root. In the workspace root it looks for a project view in .eclipse/.bazelproject file.

Our primary use case is a mono-repo & monolith. Thus, we usually open VS Code in the Bazel workspace root and use .eclipse/.bazelproject to customize the sync process. You can use directories or targets to limit what is synced into Eclipse/JDTLS & VS Code. Especially in a mono-repo you most likely only want a subset of project to be synced.

Also, can you describe the project layout? Is it more Maven like or more Bazel like?

Bazel like: verify fine grained, eg., BUILD files within Java packages to slice Java packages into smaller libs/units Maven like: all Java code in a src directory, no nested BUILD files within Java source code

For Maven like projects it may make sense to tweak the .bazelproject file a bit. Here is an example:

# Default Project View Settings for Eclipse & VS Code Users
#
# More info about this file:
#   https://github.com/salesforce/bazel-eclipse/blob/main/docs/common/projectviews.md
#
# Please do not modify this file without consulting the developer experience team.
# The file contains default settings optimized for Eclipse and VS Code and should be
# imported from .eclipse/.bazelproject file only.
#

# collapse all targets into one Java project (for Maven like layouts)
target_provisioning_strategy: project-per-package

# make the naming backwards compatible to the old Maven artifact ids,
# do not use the execution environment but hard-wire the project to runtime JDK
# reduce classpath to direct compile dependencies only (some projects have too many dependencies)
# (but be careful, can cause compile/unit test execution issues)
target_provisioning_settings:
  project_name_separator_char=-
  jre_system_library=current_java_runtime
  classpath_depth=1

# ensure Eclipse has (some) consistent settings for developers
import_preferences:
  tools/eclipse/preferences/eclipse-prefs.epf

# use same project settings everywhere
project_settings:
  tools/eclipse/project_settings/org.eclipse.core.resources.prefs
  tools/eclipse/project_settings/org.eclipse.core.runtime.prefs
  tools/eclipse/project_settings/org.eclipse.jdt.core.prefs
  tools/eclipse/project_settings/org.eclipse.jdt.ui.prefs
  tools/eclipse/project_settings/org.eclipse.ui.editors.prefs

# scan all directories for targets
derive_targets_from_directories: true

# override .bazelrc defaults for synching experience
sync_flags:
  --explicit_java_test_deps   # don't leak Bazel's old test runner deps into projects

# allow Eclipse / VS Code to discover tests by using the test classpath
test_sources:
  **/test/java

Have you tried importing into Eclipse? The output/feedback is usually better there.

[guw]

@tomasaschan Did you use the pre-release version or the latest release version? I noticed we haven't pushed a release to the stable channel in a while.

[tomasaschan]

@guw I'm using the latest stable. I can try with the preview version, but I'm not tempted at restarting this and potentially throwing away all progress...

The project is more maven-like (most sub-projects are imported from maven "polyrepos") but not everything is Java either. The .bazelproject files in most of the subprojects are outside my control; I don't own those subprojects and cannot really do much about their contents. Here's the (generated on import) .bazelproject file of my own project:

directories:
  # Add the directories you want added as source here
  path/from/monorepo/root/to/my/thing

derive_targets_from_directories: true

test_sources:
  src/test/java

targets:
  # If source code isn't resolving, add additional targets that compile it here
  # (in my file it's empty)

Do you have suggestions on what I should put there?

[guw]

I released 1.4.0 this morning. That's the version you should use. The older one was definitely broken.

Add this:

# collapse all targets into one Java project (for Maven like layouts)
target_provisioning_strategy: project-per-package

# allow Eclipse / VS Code to discover tests by using the test classpath
test_sources:
  **/test/java

The glob pattern for sources is need because folders are matched as a whole from workspace root. In general, everything in the project view file is relative from workspace root.

[tomasaschan]

I still see the same behaviors with the 1.4.0 plugin version and those additions to my .bazelproject. FWIW, the target_provisioning_strategy node is not syntax highlighted like the others - is that a very recent addition to Bazel itself? (Trying to figure out if my local bazel version would be a contributing factor here...)

[guw]

.bazelproject is not a Bazel thing but comes from the IntelliJ Bazel plug-in. We added a few things to it. https://github.com/salesforce/bazel-eclipse/blob/main/docs/common/projectviews.md

What Bazel version are you using? 7.4.1?

Did you limit the directories in the .bazelproject?

Is your project building without issues in the command line? The extension is attempting a build, too. Using the Bazel disk cache might speed things up a bit.

[tomasaschan]

Ah, I see. Yes, 7.4.1:

λ bazel version
Bazelisk version: 
> bazel version
INFO: Invocation ID: 6990bc7a-144a-459d-b664-1a8cac6793ac
Build label: 7.4.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Nov 11 21:24:53 2024 (1731360293)
Build timestamp: 1731360293
Build timestamp as int: 1731360293

And yes, a command-line bazel build :my-package-name finishes in a few seconds, successfully. .bazelproject contains a directories entry with only the current project folder (i.e. the folder where this .bazelproject file lives, which is also the folder i'm opening in VS Code) and no other entries.

It seems the plugin still tries to initialize and do things for every project in the monorepo - it takes many hours and despite having it on in the background at work all day I haven't managed to have it finish once. Every time I close and re-open the project it starts over from scratch and seems to not have anything cached from before... :/

[guw]

In that case I recommend deleting the VS Code data/workspace created by JDTLS and start over.

https://github.com/redhat-developer/vscode-java/wiki/Troubleshooting Clean the workspace directory

[tomasaschan]

That seems to have just set it back to start over again.

Not sure how much detail I can share about the repo structure and contents (there's a lot of proprietary info in there...) but I did notice the activation script started with bazel query 'buildfiles(//...) so I ran that just to give a feel for the size of our thing:

λ bazel query 'buildfiles(//...)' | wc -l 
INFO: Invocation ID: a2c5e252-198b-467f-a18c-af7025e3e267
Loading: 0 packages loaded
4242

[guw]

4242

That's ~40% of what I have here. :)

Do you see any progress in the Bazel Console in VS Code? Is there any output? Can you try loading it with Eclipse?