search

bazelbuild / intellij

IntelliJ plugin for Bazel projects
https://ij.bazel.build/
Apache License 2.0
760 stars 304 forks source link

Test indexing in Clion takes a very long time #4674

Open odisseus opened 1 year ago

odisseus commented 1 year ago

Description of the bug:

On a large C++ project, the IDE is constantly busy scanning for Google tests, Boost tests etc. This action is also quite resource-hungry, and it seems to start over after every project sync.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which Intellij IDE are you using? Please provide the specific version.

Clion 2022.3

What programming languages and tools are you using? Please provide specific versions.

C++

What Bazel plugin version are you using?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

sgowroji commented 1 year ago

Hi @odisseus, Can you provide minimal steps to reproduce the above issue. Thanks !

lukaszwawrzyk commented 1 year ago
  1. Import Bazel Project with CLion
  2. Wait for Sync to complete
  3. Wait at least 5 seconds (can take a bit longer though)
  4. Observe indexing of tests background task that slowly scans each file for each type of test
  5. Trigger sync
  6. Wait again
  7. Observe indexing of tests trigger again with no incrementality at all

However this might be just a CLion thing

dieortin commented 1 year ago

I’m also affected by this on CLion 2023.2

joshpersaud commented 8 months ago

This is still an issue. I'm on CLion 2023.3.3 It takes 4 DAYS to index my project. I let it finish once, a couple days later after a git pull and a Find Usage, it starts indexing again... it's unbearable. Constant lag and stutter because my heap get's too full from it. Constant high ram usage and cpu load.

Screenshot 2024-02-14 at 10 29 09 AM
tpasternak commented 8 months ago

We've made a small improvement to this issue in 2024.1, but it's still not fully done. In order to try it please use 2024.1 and make sure the convenience symlink directories exist before sync

mkmkme commented 6 months ago

Same here. I ended up listing all the irrelevant test directories and adding them to .idea/misc.xml semi-manually. Seems to work.

tpasternak commented 6 months ago

@mkmkme did you try 2024.1?

mkmkme commented 6 months ago

Nope, not yet. When is it expected to be released?

mkmkme commented 6 months ago

@tpasternak I've just installed 2024.1 and can say that the situation hasn't improved at all for me. The project in question is https://github.com/ClickHouse/ClickHouse. I even excluded contrib directory in order to improve it, but it seems to still take a lot of time to index.

EDIT: okay, maybe saing "hasn't improved at all" was a bit harsh, because it clearly will take only about an hour to index them. Can't say if it has to reindex all the tests after a git pull, will see. EDIT2: yes it does have to reindex everything :/

tpasternak commented 6 months ago

Btw is it a bazel project?

mkmkme commented 6 months ago

No it's a cmake project

tpasternak commented 6 months ago

For CMake issues, please report it in https://youtrack.jetbrains.com

MichaelOrlov commented 4 months ago

@tpasternak I am on CLion 2024.1.1 with a big Bazel project, and Google test indexing is continuously running forever and restarting repeatedly if stop it manually.

What did you mean by saying

make sure the convenience symlink directories exist before sync

Could you please clarify?

tpasternak commented 3 months ago

make sure the convenience symlink directories exist before sync

Could you please clarify?

Oh, this is not an issue in 2024.1

@MichaelOrlov is it possible to cut a minimal reproducible example for this?

MichaelOrlov commented 3 months ago

@tpasternak Unfortunately, in this particular case, minimum and reproducible are two contradictory entities. I think the problem is in the design of how tests are indexing in the Bazel plugin.

Imagine a Big-Big project with monorepo and many packages inside. Each package has its own BUILD.bazel file and unit tests inside. Similar to the https://github.com/orgs/ros2/repositories?type=all but each repository located in its own subfolder inside monorepo and has its own BUILD.bazel file in the same hierarchy as CMakeList.txt files.

Developers are usually interested in one of the packages for instance https://github.com/ros2/rclcpp and load relevant BUILD.bazel file via the "Partially Synch rclcpp::all" command from the Bazel plugin menu. i.e. the similar workflow as with CMakeList.txt. However, the Bazel plugin trying to reindex all tests from all packages in monorepo. Even if it takes 40-60 minutes to synch all tests for a large project. That is too long and unacceptable since during those 40-60 minutes, the developer will likely change something in the code or do a rebase or switch to another branch. Those actions will trigger all tests to re-synch and will take another 40-60 minutes. i.e., From the user perspective, it will look like an endless process of the Test indexing job.

It will make much more sense to reindex only tests referenced in the currently partially synched BUILD.bazel file. IMO, it will solve performance problems by design. This is would be a similar approach as when user loading the particular CMakeList.txt file.