Test discovery blocks VSCode Testing GUI completely

OleksandrKvl commented 1 month ago

Brief Issue Summary

In my project, I have several thousands of tests. Most of them come from around 10 GoogleTest binaries via gtest_discover_tests. Each time I save one of my CMake files, CMake: Configure step is triggered (which is fine), configuration is performed pretty quickly:

...
[cmake] -- Configuring done (0.1s)
[cmake] -- Generating done (0.1s)

but then seems that extension populates test list and this takes noticeable amount of time (around 3 seconds) and during this process VSCode GUI is completely blocked, I can't press any button or edit the code. I work on this project for a while and I'm almost sure that it didn't work like this before.

VSCode version: 1.93.1 CMake tools version: v1.19.52 OS: Darwin arm64 23.5.0

CMake Tools Diagnostics

No response

Debug Log

No response

Additional Information

No response

Amy-Li03 commented 1 month ago

Hi @OleksandrKvl, thank you for reporting issue here! We have a simple project with gtest_discover_tests try to repro your issue, unfortunately, we cannot repro it, we didn't find "VSCode GUI is completely blocked" on our end. However, we have attached our test project and repro video, please correct us if we have done something incorrect. Thank you in advanced! ctest-project.zip

NotRepro_issue4098

OleksandrKvl commented 1 month ago

@Amy-Li03 your project doesn't reproduce the problem, I think that's because you have only 2 tests there. After checking the log, I figured out that problematic command is cmake.refreshTestsAll. I've attached the video where I click Refresh Tests button and you can see that for 1-2 seconds I moved the cursor over the test list and it was not responsible (as the whole VSCode window). I'd say that the "stuck" time is proportional to the number of tests.

https://github.com/user-attachments/assets/e4ddefba-a8c8-447c-aff6-4eb9438570bb

The project is open source, you can try it on your side but it requires some extra steps to install the deps (I use Conan but anything else should work as long as find_package can find the deps). Repo: https://github.com/OleksandrKvl/sbepp Instructions: https://oleksandrkvl.github.io/sbepp/1.4.1/installation.html You'll need at least C++17, fmt, pugixml, googletest, SBEPP_DEV_MODE=ON and SBEPP_BUILD_TESTS=ON to enable the tests.

gcampbell-msft commented 1 month ago

@OleksandrKvl Thanks for reporting this, we will put it On Deck and hope to prioritize it as soon as we can

gcampbell-msft commented 1 month ago

@OleksandrKvl Before I begin investigation on this bug, could you please tell me what you have set for these settings?

cmake.ctest.allowParallelJobs cmake.ctest.testExplorerIntegrationEnabled

Thanks!

OleksandrKvl commented 1 month ago

cmake.ctest.allowParallelJobs is OFF in Workspace but ON in User settings. cmake.ctest.testExplorerIntegrationEnabled is ON in both.

OleksandrKvl commented 1 month ago

Maybe another not exactly correct behavior: test discovery is triggered even when CMake configuration (triggered by Save file) fails.

Amy-Li03 commented 4 weeks ago

Maybe another not exactly correct behavior: test discovery is triggered even when CMake configuration (triggered by Save file) fails.

Hi @OleksandrKvl , thanks for your info here! In order to better track issue, please create a separate GitHub issue for this.

gcampbell-msft commented 2 weeks ago

@OleksandrKvl Thanks for the additional information and context, I've assigned myself and hope to get to it as soon as I can

gcampbell-msft commented 2 weeks ago

@OleksandrKvl Could you confirm whether it's just the Test Explorer UI that isn't responding? Or is it the entirety of the VS Code UI?

OleksandrKvl commented 2 weeks ago

@gcampbell-msft it's the VS Code UI.

gcampbell-msft commented 2 weeks ago

@OleksandrKvl Could you provide an example or gif of that being showcased? In the video you pasted in an earlier comment, it's only the Test Explorer UI

OleksandrKvl commented 2 weeks ago

I can try to do that in future days (a bit busy right now) but I'm sure about it because that was how I noticed it at the first place. I have a habit of pressing Cmd+S frequently and doing this while editing CMake files became a big problem because the editor was unresponsive during test discovery so I had to press Cmd+S very carefully to avoid those moments.

gcampbell-msft commented 1 day ago

@OleksandrKvl I wanted to update you that I've been able to reproduce the issue and will be investigating as soon as I can, we are currently doing our best to target this for our next release. Thanks!

gcampbell-msft commented 11 hours ago

@OleksandrKvl Have you noticed this problem on Windows ever?

I'm only able to get your repro opened up on Linux/Mac, and I also am only noticing the issue on Linux/mac. Thanks

OleksandrKvl commented 11 hours ago

@gcampbell-msft no, I'm on macOS, don't know how it behaves on other systems.

gcampbell-msft commented 10 hours ago

A quick update. I've made some progress, but I don't currently have any plans on how to fix it.

It seems like it's hanging when the ctest executable is being run here https://github.com/microsoft/vscode-cmake-tools/blob/07b97938c8392b8a811972f349c6b420fcaf8cea/src/ctest.ts#L768. Unfortunately, we have to execute this command in order to get the list of tests, so I'm not currently sure of how to get around this. I'm not sure if there is a difference but I haven't been able to get this to repro on Windows.

@paulmaybee @moyo1997 @itodirel FYI.

OleksandrKvl commented 9 hours ago

Sure, some commands are required to be executed to get that list but why should it be synchronous and block the whole UI (actually looks like it's async)?

gcampbell-msft commented 8 hours ago

@OleksandrKvl Oh, I completely agree, sorry I didn't make that clear, was just trying to provide some context and update on my investigation.

As you pointed out, we're currently trying to investigate and understand why it's blocking the UI despite being async. Thanks

microsoft / vscode-cmake-tools