"Refreshing Files" step of sync takes a really long time

[keithl-stripe]

Some engineers here at Stripe are seeing IntelliJ spend many minutes performing "Refreshing files." This often happens after every Bazel IntelliJ sync.

When we pause IntelliJ in the debugger, we see that it's stuck indexing the execution root for the Bazel workspace:

(This behavior appears to have been changed – or maybe introduced – in #1703, though I'm not sure how (or why).)

On my laptop, this means it's refreshing (or reindexing?) hundreds of thousands of files:

% find /private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander | wc -l
  134063
% find -L /private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander | wc -l
  151299

Most of the CPU is spent just traversing the files:

If I set a logging-only breakpoint in com.intellij.openapi.vfs.VirtualFileVisitor.visitFileEx I see it visiting runfiles for hundreds of tests:

file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/commons/commons_lang3.jar
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/commons/commons_math3.jar
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/commons/commons_compress.jar
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/logging
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/logging/log4j
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/logging/log4j/log4j_api.jar
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/logging/log4j/log4j_core.jar
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/apache/logging/log4j/log4j_slf4j_impl.jar
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/xerial
file:///private/var/tmp/_bazel/479f88e6c4756e5120907a9f85b5f7d8/execroot/com_stripe_zoolander/external/com_stripe_uppsala/bazel-uppsala/bazel-out/darwin-fastbuild/bin/src/test/java/com/stripe/XXX/test_XXX.runfiles/com_stripe_uppsala/external/third_party_jvm/3rdparty/jvm/org/xerial/snappy

Some questions I have:

• Why is IntelliJ VFS even aware of the execroot itself? It seems that it would be much more efficient for the VFS roots to be individual jars and source paths.
  • Are the bazel-<workspacename>, bazel-out symlinks relevant? We usually use directories: . in our .bazelproject which may inadvertently be causing this problem
  • If so, should we just generate the .bazelproject file to have directories: contain a list of all subdirectories except bazel-*?
• Why do the Python and Go plugins need to refresh the execution root, but other plugins don't? Is this a bug that could be fixed? We write a good bit of Python here and would rather not disable that Bazel IntelliJ language extension.
• Could we work around this by using excludes somehow? The IntelliJ docs are not clear about whether this helps:

The IntelliJ plugin developer docs suggest we cannot just add those to some kind of ignore/exclude list:

The VFS itself does not honor ignored files listed in Settings/Preferences | Editor | File Types and folders to ignore and excluded folders listed in Project Structure | Modules | Sources | Excluded. If the application code accesses them, the VFS will load and return their contents. In most cases, the ignored files and excluded folders must be skipped from processing by higher-level code.

However the IntelliJ product docs say otherwise:

Marking dynamically generated files as excluded can speed up the indexing and overall IDE performance. For example, it's recommended that you exclude compilation output folders. Excluded files remain a part of a project, but are ignored by code completion, navigation, indexing, and inspections.

[dfinninger]

We recently added our Python code to Bazel in our monorepo (which was previously just managed Java & Scala via Bazel). Intellij now freezes when it starts refreshing files and eventually crashes due to out of memory errors. Every time this happened, I bumped the memory up by 1GB to see if that would fix it. I'm up to 6GB now and it just takes longer to crash.

To corroborate this bug report, I ran dtruss on Intellij and confirmed that when the IDE locks up it is scanning all the files in the execroot. These directories are all excluded via Intellij.

# ij.bazelproject 
directories:
  .
  -bazel-bin
  -bazel-dataworks
  -bazel-genfiles
  -bazel-out
  -bazel-testlogs

targets:
  //...:all

java_language_level: 8

additional_languages:
  scala

Since adding a hermetic Python install to Bazel, the scanning is now nearly impossible to deal with. My "workaround" is to run a bazel clean every morning and every time I switch modules. Then I "partially sync" the folder I am working on, and while it still scans the files it completes after a few minutes when just indexing a single module.

[sgowroji]

Hello @keithl-stripe & @dfinninger, Could you please confirm if you are still seeing the above performance issue even with the latest intellij and bazel plugin releases. Thanks!

[dfinninger]

I believe I'm up to date on minor versions (probably a few patches behind), and I'm still seeing the issue. However I'm on PTO this week so I won't be able to test for a little bit.

I'll upgrade next week and try it out again.

[dfinninger]

@sgowroji I've updated to the latest versions I could get in the Jetbrains Toolbox app. IDEA Version: IntelliJ IDEA 2022.2.3 (Ultimate Edition), Build #IU-222.4345.14, built on October 4, 2022 Bazel Plugin version: 2022.09.20.0.1-api-version-222

And while indexing I still see it looking through my excluded Bazel directories:

❯ sudo dtruss -p 21559 | grep _bazel_dfinninger | grep getattrlist
getattrlist("/private/var/tmp/_bazel_dfinninger/e0de62c3aa2a01df70346b0386e30cf1/execroot/REDACTED/bazel-out/darwin-fastbuild/bin/REDACTED/src\0", 0x7FF8093C41E4, 0x70000DEECEF0)               = 0 0
getattrlist("/private/var/tmp/_bazel_dfinninger/e0de62c3aa2a01df70346b0386e30cf1/execroot/REDACTED/bazel-out/darwin-fastbuild/bin/REDACTED/src/main\0", 0x7FF8093C41E4, 0x70000DEECEF0)          = 0 0
getattrlist("/private/var/tmp/_bazel_dfinninger/e0de62c3aa2a01df70346b0386e30cf1/execroot/REDACTED/bazel-out/darwin-fastbuild/bin/REDACTED/src/main/python\0", 0x7FF8093C41E4, 0x70000DEECEF0)           = 0 0
getattrlist("/private/var/tmp/_bazel_dfinninger/e0de62c3aa2a01df70346b0386e30cf1/execroot/REDACTED/bazel-out/darwin-fastbuild/bin/REDACTED/src/main/python/print.runfiles\0", 0x7FF8093C41E4, 0x70000DEECEF0)            = 0 0

... etc ...

[github-actions[bot]]

Thank you for contributing to the IntelliJ repository! This issue has been marked as stale since it has not had any activity in the last 6 months. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-maintainer". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!

[dhalperi]

@bazelbuild/triage - I'm having trouble understanding why this issue was autoclosed. What's the underlying thinking here?

The users responded promptly to every request for more data or more information, all was given, and the issue was added to Bazel backlog.

[dhalperi]

@mai93 since that alias linked in the comment above apparently did not resolve to a team.

[dhalperi]

And @sgowroji

[dhalperi]

What's the customer obsessed reason to have the bot do this when all action is pending on Bazel team?

[idanakav]

This could be fixed now with IntelliJ 2024.1 and after #6094 is merged. There is more context in IDEA-333933.

@tpasternak can you please confirm?

[tpasternak]

There are multiple factors that might impact it, but I believe the worst one is already fixed

[piyush-golani]

I am facing the same issue with IntelliJ 2024.1 my bazel sync is stuck in refreshing files and is taking more than 3 hours to complete a small sync

[tpasternak]

hey @piyush-golani

3 hours is indeed enormous size. It looks like you are working on a really large codebase. Could you try to reduce the scope by manipulating targets/directories sections in the projectview file? Also, we still don't support non-default convenience symlink locations